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PREFACE 

The  use  of  the  term  "A OP"  or  "system"  in  this  document  is  not  meant 
to  imply  any  particular  functional  category  or  system.  In  particular, 
the  term  is  meant  to  encompass  at  least  the  four  categories  outlined  in 
AFR  800-14:  Category  A--ADP  resources  in  combat  weapon  systems  and 

specially  designed  equipment;  Category  3--ADP  resources  in  other  systems 
developed  under  AFR  800-2;  Category  C--ADP  resources  in  systems 
developed,  acquired,  and  managed  by  AFR  80-2,  AFR  65-2,  AFR  71-11,  and 
AFR  100-2;  and  Category  D--ADP  resources  in  general  purpose,  ADPS 
developed,  acquired,  and  managed  by  the  300-series  regulations  and 
manuals.  Primary  application  of  risk  assessment  tools  and  methodologies 
will  be  to  mission-critical  ADP  systems  covered  by  categories  A  and  B  in 
accordance  with  AFR  800-14. 
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SECTION  I 
INTRODUCTION 


1.1  BACKGROUND. 


The  Air  Force  Operational  Test  and  Evaluation  Center  (AFOTEC)  has 
the  responsibility  for  performing  operational  test  and  evaluation  (OT&E) 
of  assets  entering  the  Air  Force  inventory.  AFOTEC  has  developed  and 
implemented  various  software  OT&E  methodologies.  These  methods  have 
matured  and  have  become  the  Air  Force  standard  for  evaluating  software 
supportabi 1 ity.  Each  of  these  developed  methods  evaluates  specific  char¬ 
acteristics  of  the  supportabi 1 ity  aspects  of  delivered  software  and  soft¬ 
ware  support  resources.  These  stand-alone  evaluations  provide  AFOTEC 
with  information  to  identify  particular  software  supportabi 1 ity  defi¬ 
ciencies,  but  do  not  identify  overall  risk  associated  with  contractor  or 
military  ownership  and  organic  maintenance  of  contractor-delivered 
software. 

Assessing  the  software  supportabi 1 ity  risk  of  Air  Force  acquired 
systems  is  necessary  to  enable  various  decision  makers  to  properly  plan 
for  system  deployment.  Risk  assessment  (RA)  is  required  throughout  the 
system  acquisition  life  cycle.  The  perspective  of  OT&E  is  focused  upon 
the  overall  system  mission  operation,  including  support.  Methods  are 
needed  to  provide  software  testers  with  areas  which  require  testing 
emphasis,  and  decision  makers  with  an  assessment  of  the  software  support¬ 
abi  1 ity  risk. 

Software  support  for  major  weapon  systems  is  becoming  a  major  system 
cost  factor.  Major  weapon  systems  are  using  more  sophisticated  computer 
systems  and  the  support  costs  required  for  embedded  software  is  projected 
to  increase.  Furthermore,  since  most  enhancements  to  the  system  are 
dependent  on  software  modifications,  the  timeliness  of  such  software 
support  is  critical  to  system  operational  availability  and  effectiveness. 
Because  of  this  criticality  of  the  software  support  function  to  overall 
system  mission  operational  capability,  it  is  desired  that  top  decision 
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makers  be  aware  of  the  risk  associated  with  the  software  supportabi I i ty 
of  a  system  at  the  conclusion  of  OT&E.  In  order  to  determine  this  risk 
during  OT&E,  AFOTEC  needs  to  develop  and  implement  a  risk  assessment 
model  of  software  supportabi 1 ity  with  the  proper  system  mission  perspec¬ 
tive  to  ultimately  assist  the  top  level  decision  maker.  Due  to  the  com¬ 
plexity  of  this  requirement,  it  is  first  necessary  to  determine  the 
feasibility  of  developing  and  implementing  such  a  model. 

AFOTEC  produced  a  concept  proposal  (reference  5.12)  for  computer 
resources  risk  assessment  during  operational  test  and  evaluation.  This 
effort  integrates  an  approach,  appropriate  models,  and  subjective  and 
quantitative  software  operational  and  supportabi 1 ity  measures  into  a 
management-oriented  assessment  of  user  and  supporter  risk.  This  initial 
involvement  with  the  application  of  risk  assessment  to  software  support- 
ability  provided  AFOTEC  with  justification  to  support  a  stjdy  of  tie 
feasibility  of  developing  and  implementing  a  risk  assessment  model  for 
software  supportabi 1 ity  (RAMSS).  The  AFOTEC  Subtask  304  (reference  5.0) 
is  the  statement  of  this  feasibility  study's  objectives  and  required 
reports.  This  report  documents  one  part  of  this  study. 

1.2  STUOY  OBJECTIVE. 

The  overall  objective  of  this  task  study,  as  stated  in  Subtask 
Statement  304/00  (reference  5.0),  is  to  perform  a  feasibility  study  to 
determine  the  level  of  effort  and  usefulness  of  developing  and  imple¬ 
menting  a  risk  assessment  model  for  software  supportabi 1 ity  (RAMSS).  The 
report  of  reference  5.31  documents  the  first  part  of  the  effort:  to 
"review  defense  and  technical  literature  and  current  research  concerning 
methods  of  software  supportabi 1 ity  testing  and  risk  assessment  applicable 
to  an  OT&E  environment"  (reference  5.0). 

The  emphasis  for  the  first  part  of  the  task  was  placed  upon: 
a)  Identifying  and  collecting  information 

1)  Literature  search  and  review 

2)  Fact-finding  visits/conference 

3)  Contact  with  risk  assessment/software  experts 
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b)  Assembling  risk  assessment  data  base 

1)  Glossary  of  terms 

2)  Annotated  bibliography 

3)  Key  documents 

4)  Experts/knowledgeable  contacts  list 

5)  Current  research  list. 

This  report  documents  the  second  part  of  the  overall  task  study: 
"based  on  the  literature  and  research  review,  analyze  the  feasibility  of 
developing  and  implementing  a  RAMSS  that  could  be  applied  to  the  military 
systems  during  AFOTEC-conducted  OT&E"  (reference  5.0).  The  emphasis  for 
the  second  part  of  the  task  was  placed  upon: 

a)  Analyzing  current  literature  and  research 

1)  OTIC,  NTIS ,  NBS,  RADC,  AFOTEC,  DoD,  periodicals,  etc. 

2)  Potential  RAMSS,  or  parts  of  RAMSS 

3)  Continued  contact  with  risk  assessment/software 
experts 

b)  Developing  a  potential  framework  for  a  feasible  RAMSS 

1)  RAMSS  framework 

2)  Risk  assessment  methodologies,  techniques,  tools. 

1.3  STUDY  APPROACH. 

A  three-step  study  approach  was  adopted  in  Subtask  Statement  304/00. 
The  steps  were: 

a)  Conduct  a  literature  search  and  research  review. 

b)  Analyze  the  literature  and  research  information  to  deter¬ 
mine  the  feasibility  of  developing  and  implementing  a 
RAMSS  to  be  applied  to  military  systems  during  AFOTEC- 
conducted  OT&E. 

c)  Identify  and  analyze  candidate  measures  of  supportabi 1 ity 
risk  for  use  in  developing  a  feasible  RAMSS. 

The  first  step  results  are  presented  in  the  report  of  reference 
5.31.  The  literature  search  and  review  required  identification  of  key 
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documents  published  by  governmental  agencies  and  civilian  agencies. 
Literature  searches  of  the  Defense  Technical  Information  Center  (OTIC), 

National  Technical  Information  Service  (NTIS),  and  Rome  Air  Development 
Center  (RADC)  data  bases  were  conducted.  A  search  and  review  of  National 
Bureau  of  Standards  (N8S)  publications  was  done.  Key  documents  from 
these  searches  were  identified  and  ordered  for  inclusion  in  the  RA  data 
base.  Several  documents  from  another  AFOTEC  subtask  294  on  Computer 
System  Security  (reference  5.32)  were  identified.  The  final  report 
bibliography  will  include  any  additional  documents  selected  from  that 
list.  Researching  the  available  RA  technology  also  involved  contact  with 
a  number  of  agencies,  and  identification  of  and  discussions  with  RA 
research  and  evaluation  personnel.  The  basic  form  and  content  of  this 
data  base  of  RA  information  is  described  in  reference  5.31.  but  will  be 
augmented  and  updated  as  necessary  to  keep  the  data  base  current  through¬ 
out  this  study. 

The  second  step  results  are  presented  in  this  report.  Analysis  of 
candidate  RAMSS  involved  analysis  of  literature  and  research  collected  • 
from  step  1  in  the  two  areas  of  risk  assessment  and  software  support- 
ability.  Very  little  crossover  between  the  two  areas  was  evident. 

Hence,  it  was  important  for  the  feasibility  requirement  of  this  step  to 
analyze  the  elements  of  risk  assessment,  factors  of  software  support- 
ability,  and  develop  a  framework  within  which  it  could  be  determined 
whether  these  "pieces"  of  a  RAMSS  could  be  integrated  and  implemented  as 
a  RAMSS. 

1.4  REPORT  ORGANIZATION. 

The  remainder  of  this  report  is  organized  into  five  sections  plus  a 
set  of  appendices  that  include  the  detailed  information  concerning  the 
activities  described  in  paragraph  1.3.  Report  sections  satisfy  the 
following  objectives: 

a)  Section  II  contains  a  summary  of  the  analysis  conducted, 

candidate  RAMSSs,  level  of  effort  to  develop  and  implement  -.T\ 
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candidate  RAMSSs,  a  framework  for  a  RAMSS,  and  potential 
risk  assessment  methodologies,  techniques,  and  tools. 
Section  III  contains  the  technical  details  of  a  foundation 
framework  within  which  an  RAMSS  could  be  developed  and 
implemented.  This  section  focuses  on  terminology, 
criteria,  and  constraints  for  such  a  model,  a  software 
supportability  risk  management  framework  for  QT&E,  a 
structure  for  the  software  supportabil ity  risk  management 
process,  potential  measures  of  software  supportability 
risk,  and  methods  of  reporting  results  of  the  risk  assess¬ 
ment  process.  . 

Section  IV  contains  technical  details  on  risk  assessment 
methodologies,  techniques  and  tools  which  might  support 
development  and  implementation  of  an  RAMSS.  The  theoret¬ 
ical  foundation  of  risk  assessment  is  briefly  reviewed. 
Subjective  and  objective  methodologies,  techniques,  and 
tools  are  described  in  enough  detail  so  the  nature  of 
their  applicability  to  an  RAMSS  can  be  better  understood. 
Section  V  lists  the  documents  whose  contents  have  been 
referenced  in  this  report. 

Appendix  A  lists  acronyms  used  in  this  report. 

Appendix  B  is  a  glossary  of  terms  (sources  of  the  terms 
and  descriptions  are  listed)  used  in  this  report. 

Appendix  C  is  a  summary  of  DoD  and  Air  Force  policy  and 
directives  concerning  risk  management. 
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SECTION  II 
EXECUTIVE  SUMMARY 

2.1  INTRODUCTION. 

This  section  of  the  report  provides  an  overview  of  the  material 
presented  in  sections  III  and  IV.  This  overview  summarizes  the  analysis 
conducted,  lists  candidate  Risk  Assessment  Models  for  Software  Support- 
ability  (RAMSSs),  discusses  the  level  of  effort  to  develop  and  implement 
candidate  RAMSSs,  describes  an  initial  framework  for  an  RAMSS,  and 
examines  potential  risk  assessment  methodologies,  techniques,  and  tools. 
Also  included  are  the  basic  conclusions  drawn  from  the  study  and  analysis 
of  the  literature  and  research  performed  during  the  first  phase  of  this 
subtask. 

The  reader  is  referred  to  appendix  C  of  this  report  for  a  summary  of 
material  from  directives  of  higher  authorities  and  the  military  services. 
#  This  material  supports  the  need  for  the  performance  of  a  risk  assessment 

study. 

2.2  ANALYSIS  CONDUCTED. 

The  material  analyzed  during  this  study  included  documents  obtained 
from  the  Defense  Technical  Information  Center  (OTIC);  the  Rome  Air  Devel¬ 
opment  Center  (RADC);  the  National  Technical  Information  Service  (NTIS); 
Risk  Analysis  (RA)  experts  and  knowledgeable  personnel  contacted  by  tele¬ 
phone,  on  fact-finding  trips  and  at  conferences;  and,  references  in  key 
documents.  The  first  report  (reference  5.31)  of  this  subtask  contains 
the  list  of  documents  and  sources  which  were  used  as  a  data  base  for  this 
study.  At  the  time  of  the  current  report  (August  31,  1984),  the  major 
sources  of  documents,  and  the  document  counts,  are  given  in  table  2-1. 
The  Computer  System  Security  (CSS)  task  listed  below  includes  data  from 
reference  5.33. 
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Table  2-1. 

RA  Data  Base  Summary 


Source  of  Data 

Quantity  of 

Documents 

Identified 

Quantity  of 

Documents 

Ordered 

DTIC  (1970-1984) 

450 

5 

NTIS  (1964-1984) 

3000 

53 

RADC 

3200 

21 

CSS  TASK 

16 

16 

AFOTEC 

13 

13 

OTHER/IN  HOUSE 

65 

65 

6744 

173 

Whereas  a  large  number  of  documents  were  identified  via  the  litera¬ 
ture  search  on  key  words,  it  was  found  that  a  relatively  small  number  of 
documents  actually  applied  to  the  subject  matter  at  hand.  8DM  personnel 
have  obtained  one-third  of  the  total  documents  from  other  (experts, 
references  in  key  documents,  etc.)  or  in-house  sources.  Of  the  total  of 
173  documents,  approximately  one-fourth  of  them  have  been  identified  as 
"key"  documents,  in  the  sense  that  these  documents  contained  information 
which  was  directly  pertinent  to  the  study  of  risk  assessment  of  software 
supportabil ity  in  the  OT&E  environment.  These  documents  are  listed 
separately  in  section  V  of  this  report,  and  form  the  basis  for  much  of 
the  analysis  performed. 

2.3  SCOPE  OF. CANDIDATES  FOR  AN  RAMSS. 

From  the  entire  literature  and  research  review,  only  one  RAMSS  has 
been  found.  This  model  is  described  by  F.  Fisk  and  W.  Murch  in  reference 
5.12,  and  is  well  known  to  AFOTEC.  The  proposed  Fisk/Murch  model  is 
preliminary  and  is  not  officially  sanctioned  by  AFOTEC.  The  importance 
of  this  model  is  its  view  of  evaluating  and  reporting  software  user  and 
supporter  risks  associated  with  the  acceptance  of  computer  resources, 
especially  software.  The  model  framework  integrates  aspects  of  current 
AFOTEC  developed  methodologies  for  evaluating  computer  resources,  without 
restricting  the  possibility  of  including  other  methodologies. 
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One  other  model  is  currently  being  developed  by  Georgia  Tech 
(reference  5.39).  At  the  time  of  this  report,  the  model  is  still  in  the 
conceptual  stage.  The  Georgia  Tech  model  is  essentially  a  top  down 
approach  based  upon  decision  theory. 

Further  details  of  these  models,  along  with  advantages  and  limita¬ 
tions,  are  presented  in  section  IV  of  this  report. 
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2.4  LEVEL  OF  EFFORT  FOR  RAMSS  DEVELOPMENT  AND  IMPLEMENTATION. 


>1 


1 


A  major  conclusion  of  the  analysis  is  that  it  is  probably  feasible 
to  develop  and  implement  an  RAMSS.  Other  than  the  proposed  Fisk/Murch 
model  and  the  Georgia  Tech  model,  no  current  model  exists.  Therefore,  a 
candidate  RAMSS  would  have  to  be  either  one  of  these  models  or  a  combina¬ 
tion  of  the  techniques  listed  in  section  2.6,  and  fall  within  the  general 
RAMSS  framework  proposed  in  section  III.  There  appear  to  be  enough 
methods  for  risk  assessment  that,  when  the  advantages  and  limitations  of 
each  methodology  or  technique  are  compared,  either  one  or  some  combina¬ 
tion  will  be  selected  for  development  and  implementation.  The  actual 
comparison  and  selection  of  these  techniques  will  be  completed  in  the 
next  stage  of  this  subtask. 

A  preliminary  examination  of  the  development  and  implementation  of 
an  RAMSS  indicates  that  the  task  may  be  better  accomplished  in  three 
phases.  The  first  phase  consists  of  the  work  currently  undertaken,  which 
consists  basically  of  a  concept  development.  The  second  phase  should 
constitute  a  development  of  model  requirements,  a  checkout  of  the 
selected  procedures,  and  a  model  design.  The  third  phase  should  consist 
of  a  review  of  the  model  concept  based  upon  information  obtained  from 
phase  two,  followed  by  the  actual  development  and  implementation  of  the 
model.  The  estimates  of  the  resources  required  to  complete  these  phases 
are: 
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Resources 

Phase  Duration  (Months)  Staffing  (People) 

I  5  3.5 

II  6  3 

III  8  4 

See  section  3.6  for  more  details  about  these  preliminary  estimates. 

2.5  FRAMEWORK  FOR  AN  RAMSS. 

It  is  possible  to  develop  a  framework  for  an  RAMSS  which  bridges  the 
gap  between  the  theoretical  aspects  of  risk  assessment  and  the  applica¬ 
tion  of  risk  to  OT&E  software  supportability.  This  framework  takes  into 
account  such  factors  as:  identifying  risk  agents,  determining  negative 
outcomes,  estimating  probability  of  negative  outcomes,  reducing  risk  and 
choosing  alternatives,  accepting  risks,  and  evaluating  uncertainty. 
These  factors  are  built  into  a  framework  based  on  risk  determination, 
which  is  the  process  of  identifying  and  estimating  the  magnitude  of  the 
risk,  and  risk  evaluation,  which  is  the  complex  process  of  determining 
acceptable  levels  of  risk  and  alternative  risk  choices. 

The  framework  factors  discussed  above  must  also  be  measured  in  order 
to  build  an  effective  model.  The  measures  determined  most  applicable  to 
the  software  supportability  risk  assessment  process  include:  embedded 
computer  system  profile  metrics,  evaluation  metrics,  negative  outcome 
probability  estimates,  magnitude  of  consequence  estimates,  risk  levels, 
and  risk  agent  acceptance  levels.  Some  of  these  measures,  such  as  the 
evaluation  metrics,  are  already  being  captured  by  AFOTEC  (see  reference 
5.1) ,  but  many  are  not. 

Details  of  the  proposed  framework  may  be  found  in  section  III  of 
this  report. 

2.6  RISK  ASSESSMENT  METHODOLOGIES,  TOOLS,  TECHNIQUES. 

There  are  several  techniques  for  risk  assessment  which  have  not  been 
married  into  one  model  for  an  assessment  of  software  development,  but 
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which  could  be  considered  to  assist  in  the  development  and  implementation 
of  an  RAMSS.  These  techniques,  which  could  also  be  referred  to  as 
methodologies  or  tools  as  they  often  are  in  the  literature,  are  as 
follows: 

a)  Choice-between-gambles  technique 

b)  Standard  lottery 

c)  Modified  Churchman-Ackoff  technique 

d)  Delphi  procedure 

e)  Closed  form  questionnaires 

f)  Bayesian  analysis 

g)  Network  analysis 

h)  Decision  trees 

i)  Parametric  model ing 

j)  Decision  theory 

These  techniques  may  be  grouped  as  subjective,  objective,  or  some 
combination  of  both.  Since  it  is  not  uncommon  for  entire  textbooks  to  be 
devoted  to  any  of  these  ten  topics,  it  will  be  the  intent  of  this  report 
to  give  the  reader  only  a  flavor  of  the  technique's  mechanics,  advan¬ 
tages,  and  limitations.  Such  a  discussion  is  found  in  section  IV  of  this 
document. 


2.7  CONCLUSIONS. 
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The  basic  conclusions  from  the  analysis  of  the  literature  and 
research  data  are: 

a)  No  directly  applicable  RAMSS  exists. 

b)  There  is  very  little  research  in  risk  assessment  and  soft¬ 
ware  where  the  efforts  are  integrated,  with  the  possible 
exception  in  CSS,  as  reported  in  reference  5.34. 

c)  The  closest  model  to  an  RAMSS  is  the  proposed  Fisk/Murch 
model.  It  is  feasible  to  implement  such  a  model,  but 
there  are  serious  limitations  in  the  theoretical  founda¬ 
tion  as  discussed  in  section  4.4.2. 
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The  RAMSS  framework  presented  in  section  III  and  the  tech¬ 
niques  reviewed  in  section  IV  give  reasonable  credibility 
to  claim  that  it  would  be  feasible  to  develop  and  imple¬ 
ment  an  RAMSS. 
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SECTION  III 
RAMSS  FRAMEWORK 


3.1  TERMINOLOGY  AND  FOCUS. 


This  section  provides  a  pragmatic  bridge  between  the  theoretically- 
based  aspects  of  risk  assessment  methodologies,  techniques,  and  tools  and 
the  subject  application  area  of  software  supportabi 1 ity  OT&E.  The  theo¬ 
retical  foundation  (set  in  probability  theory)  of  risk  and  several  risk 
estimation  methodologies/techniques  are  discussed  in  section  IV.  A  risk 
management  framework  and  the  elements  of  software  supportabi! ity  from 
which  a  feasible  Risk  Assessment  Model  for  Software  Supportabi 1 ity 
(RAMSS)  could  be  developed  are  presented  in  this  section.  Much  of  this 
information  is  an  expansion  of  material  in  section  IV  of  the 
reference  5.31. 

The  key  terms  to  understand  in  defining  a  risk  assessment  model  for 
software  supportabi 1 ity  include  risk,  model,  and  software  supportabi 1 ity . 
Software  supportabi 1 ity  is  the  subject  of  the  assessment.  Risk  is  what 
is  determined  by  the  assessment.  And,  the  complete  process  as  well  as 
the  descriptive  characteristics  of  software  supportabi 1 ity  constitute  an 
approximation  to  reality;  that  is,  a  model. 

What  the  assessment  process  should  produce  is  a  measure  of  the 
potential  that  support  for  a  specific  software  product  (or  set  of  pro¬ 
ducts)  will  not  satisfy  requirements.  This  potential  is  represented  as  a 
probability  and  is  "determined"  for  each  of  the  possible  negative  out¬ 
comes  (i.e.,  requirement  or  set  of  conditions  which  is  defined  to  be 
unsatisfactory).  Once  this  probability  is  determined  for  each  negative 
outcome,  thus  creating  a  probability  density  function,  risk  can  be 
determined  by  summing  (or  integrating)  over  the  appropriate  class  of 
negative  outcomes  to  arrive  at  the  risk  for  that  specified  class. 
Integrating  over  al 1  defined  negative  outcomes  determines  the  risk  for 
software  supportabi 1 ity. 

A  model  for  the  risk  assessment  process  discussed  above  will  come  as 
close  as  possible  to  representing  the  real  world.  However,  it  is 
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important  for  the  reader  to  realize  that  all  the  characteristics  of  soft¬ 
ware  supportability  which  might  determine  whether  requirements  are  met 
cannot  be  determined.  All  the  possible  negative  outcomes  cannot  be 
determined.  The  potential  for  negative  outcomes  cannot  be  determined 
with  exact  precision.  This  is  the  nature  of  a  model .  A  model  has 
uncertainty.  Thus,  there  is  some  uncertainty  in  the  resulting  risk 
assessment.  It  is  important  to  be  able  to  estimate  the  bounds  for  this 
uncertainty.  This  is  part  of  a  model  validation  and  helps  determine  the 
confidence  with  which  one  can  use  the  model.  These  limitations  do  not, 
however,  mean  that  models  are  not  beneficial,  but  that  one  must  under¬ 
stand  what  is  being  modeled. 

Some  of  the  more  important  software  supportability  related  terms  are 
defined  in  table  3-1.  Some  of  the  more  important  risk  related  terms  are 
defined  in  table  3-2.  Other  terms  are  defined  in  the  Glossary 
(appendix  B) . 

3.2  CRITERIA  AND  CONSTRAINTS. 

Software  supportability  encompasses  the  personnel,  resources,  and 
procedures  necessary  to  assure  that  software  can  be  installed,  operated, 
and  modified  to  meet  user  requirements  within  acceptable  limits.  The 
OT&E  of  software  and  software  support  resources  by  the  Air  Force  is  a 
relatively  new  effort.  The  wide  range  of  systems  containing  software, 
the  criticality  of  those  systems  to  national  defense,  and  the  ever 
present  problem  of  limited  OT&E  resources  set  the  broad  boundaries  of  the 
general  risk  assessment  criteria  and  constraints.  The  difference  can  be 
rather  significant  between  the  required  objectives  of  software  support- 
ability  OT&E  risk  assessment,  and  the  capability  of  AFOTEC  and  other 
designated  resources  to  accomplish  a  timely  assessment  of  adequate  depth 
and  understanding  to  assist  the  appropriate  decision  makers.  Therein 
lies  the  general  problem  statement:  Is  it  feasible  for  AFOTEC  with  their 
limited  resources  to  assess  the  risk  of  software  supportability  across 
the  wide  range  of  systems  entering  the  Air  Force  inventory  such  that  the 
assessment: 
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Table  3-1. 

Software  Supportabi 1 ity  Definitions 


SOFTWARE: 

THE  PROGRAMS  WHICH  EXECUTE  IN  A  COMPUTER.  THE  DATA  INPUT.  OUTPUT.  CONTROLS  UPON  WHICH  PROGRAM 
EXECUTION  OEPENOS  ANO  THE  DOCUMENTATION  WHICH  DESCRIBES.  IN  A  TEXTUAL  MEDIUM.  DEVELOPMENT  AND 
MAINTENANCE  OP  THE  PROGRAMS. 

SOFTWARE  FAULT: 

THE  PRESENCE  OR  ABSENCE  OF  THAT  PART  OF  A  SOFTWARE  PRODUCT  WHICH  CAN  RE  SULT  IN  SOFTWARE  FAILURE 

SOFTWARE  MAINTENANCE: 

THOSE  ACTIONS  REQUIRED  FOR: 

(1)  CORRECTION.  REMOVAL.  CORRECTION  OF  SOFTWARE  FAULTS 

(2)  ENHANCEMENT.  AOOITION/OCLETION  OF  FEATURES  FROM  THE  SOFTWARE 

(3)  CONVERSION  MODIFICATION  OF  THE  SOFTWARE  BECAUSE  OF  ENVIRONMENT  (OATA  HARDWARE)  CHANGfS 
SOFTWARE  MAINTAINABILITY : 

A  QUALITY  OF  SOFTWARE  WHICH  REFLECTS  the  EFFORT  REQUIRED  TO  PERFORM  SOFTWARE  MAINTENANCE  ACTIONS 
SOFTWARE  MAINTENANCE  ENVIRONMENT 

AN  INTEGRATION  OF  PERSONNEL  SUPPORT  SYSTEMS  ANO  PHYSICAL  FACILITIES  FOR  THE  PURPOSE  OF  MAINTAINING 
SOFTWARE  PROOUCTS. 

SOFTWARE  MAINTENANCE  MEASURES: 

MEASURES  OF  SOFTWARE  MAINTAINABILITY  ANO  ENVIRONMENT  CAPABILITIES  TO  SUPPORT  SOFTWARE 
MAINTENANCE  ACTIVITY. 

SOFTWARE  MANAGEMENT: 

THE  POLICY,  METHOOOLOGV.  PROCEDURES.  ANO  GUIDELINES  APPLIED  IN  A  SOFTWARE  ENVIRONMENT  TO  THE  SOFTWARE 
DEVELOPMENT, MAINTENANCE  ACTIVITIES.  ALSO.  THOSE  PERSONNEL  WITH  SOFTWARE  MANAGEMENT  RESPONSIBILTIES 

SOFTWARE  SUPPORT  F AC1UTY  (SSF): 

THE  FACILITY  WHICH  HOUSES  ANO  PROVIOES  SERVICES  FOR  THE  SUPPORT  SYSTEMS  ANO  PERSONNEL  REQUIRED  TO 
MAINTAIN  THE  SOFTWARE  FOR  A  SPECIFIC  EMBEDOEO  COMPUTER  SYSTEM. 

SOFTWARE  SUPPORT  ABILITY : 

A  MEASURE  OF  THE  ADEQUACY  OF  PERSONNEL.  RESOURCES.  ANO  PROCEDURES  TO  FACILITATE 

(1)  MODIFYING  ANO  INSTALLING  SOFTWARE 

(2)  ESTABLISHING  AN  OPERATIONAL  SOFTWARE  BASELINE 

(3)  MEETING  USER  REQUIREMENTS. 


Table  3-2. 

Risk  Management  Definitions 


RISK  IDENTIFICATION: 

THE  POTENTIAL  FOR  REALIZATION  OF  UNWANTED.  NEGATIVE  CONSEOUENCES  OF  AN  EVENT 
RISK  DETERMINATION: 

THE  PROCESS  OF  IDENTIFYING  ANO  ESTIMATING  THE  MAGNITUOE  OF  RISK 
'RISK(E  VALUATION: 

THE  PROCESS  OF  DEVELOPING  ACCEPTABLE  LEVELS  OF  RISK  TO  INDIVIDUALS  OR  SOCIETY. 

RISK  ASSESSMENT: 

THE  TOTAL  PROCESS  OF  QUANTIFYING  A  RISK  ANO  FINDING  AN  ACCEPTABLE  LEVEL  OF  THAT  RISK  FOR  AN  INDIVIDUAL, 
GROUP.  OR  SOCIETY.  IT  INVOLVES  BOTH  RISK  DETERMINATION  AND  RISK  EVALUATION. 

RISK  MANAGEMENT: 

THE  TOTAL  PROCESS  OF  IDENTIFYING.  CONTROLLING.  ANO  MINIMIZING  UNCERTAIN  EVENTS 
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a)  Has  a  technical  depth  and  result  format  appropriate  to 
adequately  assist  decision  makers; 

b)  Integrates  at  least  the  current  AFOTEC  evaluation  method¬ 
ologies; 

c)  Has  enough  accuracy  and  repeatability  to  warrant  confi¬ 

dence  in  its  results; 

d)  Is  based  upon  a  sound  theoretical  software  and  risk 

assessment  foundation; 

e)  Allows  for  determination  of  what  acceptable  level  of  risk 

means  depending  upon  the  identity  of  the  risk  agent  and 

the  software  supportabi 1 ity  requirements ;  and 

f)  Is  simple  to  use. 

The  AFOTEC  evaluation  constraints  under  which  the  above  criteria 
must  be  applied  include; 

a)  Resource  limitations 

1)  Personnel 

2)  Time 

3)  Oata  collection  (availability  and  accuracy) 

b)  Variable  environment 

1)  Computer 

2)  Software 

3)  Development 

4)  Testing/test  coverage  scenario 

c)  Evaluation  repeatabi 1 ity  and  understandabi 1 i ty 

1)  Evaluator  experience 

2)  Evaluation  reliability 

3)  Depth  of  evaluation  MOEs 

d)  Internal  charter 

1)  Restricts  certain  overlap  areas  (R&D) 

2)  Early  life  cycle  involvement  not  well  defined 

One  of  the  major  problems  (reference  5.14)  of  software  support- 
ability  is  the  diversity  of  software  product  and  environment  "forms"  that 
any  given  organization  must  support.  Software  source  may  be  written  in 
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several  different  languages  (even  for  one  application  system).  The 
target  operational  system  may  have  several  different  processors.  The 
development  environment  and  conf iguration  management  vary  greatly  across 
applications  and  are  frequently  not  deliverable  to  the  target  maintenance 
organization,  which  is  usually  tasked  with  supporting  several  applica¬ 
tions.  Even  when  there  is  some  early  planning  for  software  maintenance 
to  ease  such  transition  diversity,  the  "styles'1  of  software  structure  and 
programming  tend  to  vary  within  and  across  application  systems.  The  DoO 
concept  (references  5.15,  5.16)  of  one  language  (Ada)  and  a  reasonably 
uniform  support  (development  and  maintenance)  environment  (APSE)  may  help 
lessen  the  diversity  of  future  weapon  systems  support  requirements. 

The  measures  of  software  supportabi 1 ity  are  determined  from  the 
characteristics  of  the  identified  elements  and  actual  software  support 
activity  (e.g.,  the  measures  of  resources  consumed  during  software  main¬ 
tenance).  These  measures  must  be  reasonably  accurate,  easy  to  collect, 
and  based  upon  a  viable  software  supportabi 1 ity  conceptual  framework  (or 
model).  The  scale  of  measurement  must  be  consistent  across  the  charac¬ 
teristics. 

The  model /conceptual  framework  of  the  software  and  its  support 
environment,  which  represent  the  characteristics  to  be  evaluated  as  -part 
of  the  risk  assessment  process,  must  be  simple,  yet  have  reasonable 
fidelity.  The  framework  should  allow  for  evaluations  to  be  conducted 
under  varying  resource  constraints  and  test  objectives  (e.g.,  at  high 
level  or  more  detailed  level  characteristics). 

The  outcome  of  a  software  supportabi 1 ity  risk  assessment  should  be 
representable  in  a  form  which  pinpoints  high  risk  drivers  as  well  as  the 
^associated  detailed  risk  assessment  and  evaluation  information  which 
determines  why  those  drivers  are  a  high  risk.  It  is  useful  if  such 
information  can  be  organized  so  that  succeedingly  greater  detail  can  be 
derived  depending  upon  the  decision  maker  requirements. 

As  an  example,  it  should  be  possible  to  determine  the  overall  level 
of  the  supportability  risk  for  a  delivered  software  system.  If  needed, 
it  should  also  be  possible  to  determine  what  level  of  risk  is  associated 
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with  the  delivered  software  products  and  the  software  support  environ¬ 
ment.  It  may  be  necessary  to  pinpoint  the  risk  to  greater  levels  of 
depth  in  some  cases;  for  example,  to  the  level  of  identifying  which  soft¬ 
ware  modules  are  the  high  risk  drivers  or  whether  the  support  environment 
personnel,  support  systems,  and/or  facilities  are  the  high  risk  drivers. 
And  it  should  be  possible  to  obtain  risk  assessment  across  groups  of 
quality  characteristics.  For  example,  it  may  be  that  evaluation  informa¬ 
tion  indicates  the  software  is  very  reliable,  but  is  not  easily  modified 
or  able  to  be  ported  to  a  different  environment.  If  the  user  require¬ 
ments  during  deployment  of  the  system  are  likely  to  include  any  major 
modifications  or  a  conversion  to  a  new  hardware  system,  then  the  risk 
assessment  should  be  capable  of  appropriately  identifying  these  software 
support  risk  drivers. 

Risk  assessment  of  software  supportabi 1 ity  also  must  be  sensitive  to 
the  risk  agent.  The  risk  agent  may  be  the  developer,  system  user,  the 
supporter,  the  evaluator,  or  even  an  indirect  agent  such  as  the  general 
public.  The  perspective  may  vary  a  great  deal  from  one  agent  to  the 
next.  Generally,  all  agents  have  some  involvement,  and  if  anyone  has  too 
much  software  support  risk,  even  if  it  is  only  "perceived",  then  the 
other  agent's  risk  is  affected  in  a  "real"  way. 

The  bottom  line  to  the  decision  maker  will  be  whether  the  associated 
software  supportabi 1 ity  risk  is  acceptable  as  it  relates  to  system  per¬ 
formance  (user)  and  support  resource  cost  (supporter). 

3.3  SOFTWARE  SUPPQRTABIIITY  RISK  MANAGEMENT  FRAMEWORK. 

The  feasibility  of  developing  and  implementing  an  RAMSS  depends  upon 
having  an  integrated  framework  for  risk  assessment  and  software  support- 
ability.  This  section  provides  guidance  on  what  that  framework  should 
include.  First,  the  process  of  software  supportabi 1 i ty  risk  assessment 
is  examined,  using  AFOTEC  software  supportabi 1 i ty  terminology.  Second, 
the  framework  for  a  software  supportabi 1 ity  evaluation  is  reviewed.  Most 
of  this  framework  is  currently  in  use  by  AFOTEC.  Some  additional  soft¬ 
ware  support  management  factors  are  suggested.  Third,  elements  of  a 
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proposed  risk  management  model  are  introduced,  integrating  the  software 
supportability  evaluation  with  a  more  generic  approach  to  risk 
assessment. 

This  section,  in  combination  with  section  IV,  which  describes  more 
specific  details  of  potential  risk  assessment  techniques,  enables  the 
reader  to  see  various  possible  approaches  to  developing  and  implementing 
an  RAMSS.  These  sections  do  not  provide  details  of  that  development  and 
implementation  since  the  scope  of  this  report  is  limited  to  a  feasibility 
analysis.  Since  there  are  no  directly  applicable  RAMSSs  and  only  one 
approach  (see  section  4.4.2  and  reference  5.12)  which  combines  aspects  of 
risk  assessment  and  software  supportability,  the  analysis  approach  pre¬ 
sented  here  was  adopted. 

3.3.1  The  Software  Supportability  Risk  Assessment  Process. 

A  more  structured  view  of  risk  analysis/assessment  will  be  presented 
in  section  3.3.2.  The  process  described  in  this  part  includes  the 
following  aspects  tailored  to  the  software  supportability  terminology: 

a)  Identifying  risk  agents 

b)  Determining  negative  outcomes 

c)  Estimating  probability  of  negative  outcome  occurrences  and 
magnitude  of  consequence  value 

d)  Reducing  risk  and  choosing  alternatives 

e)  Acceptance  of  risk 

f)  Uncertainty. 

3. 3. 1.1  Identifying  Risk  Agents. 

Any  determination  of  risk  is  relative  to  a  particular  risk  agent. 
The  identified  risk  agents  which  may  be  involved  with  an  RAMSS  include: 
supporter,  user,  developer,  and  evaluator.  The  proposed  Fisk/Murch  risk 
model  identifies  the  user  and  supporter  risk  agents  as  primary  concerns 
for  OT&E.  The  developer  as  a  risk  agent  could  be  significant  if  aspects 
of  support  management  (as  suggested  in  section  3.3.2)  can  be  incorporated 
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into  the  software  supportabi 1 ity  evaluation  process.  It  is  clear  that 
providing  functional  capabilities  in  the  software  during  development  in 
order  to  reduce  supporter  risk  may  well  create  a  higher  developer  risk  in 
delayed  schedule,  excessive  cost,  or  technological  stress. 

3. 3. 1.2  Determining  Negative  Outcomes. 

Since  risk  is  the  potential  for  the  realization  of  negative  out¬ 
comes,  these  negative  outcomes  for  software  supportabi 1 i ty  should  be 

clearly  understood.  Negative  outcomes  are  relative  to  an  identified  risk 
agent  (supporter,  user,  developer,  evaluator).  Possible  negative  out¬ 
comes  for  each  of  the  risk  agents  can  be  determined  from  an  embedded  com¬ 
puter  system  (ECS)  software  maintenance  profile.  Negative  outcomes  occur 
when  software  supportabi 1 ity  resources  cannot  satisfy  a  particular 
software  support  requirement.  The  software  support  requirements  are 

based  upon  the  ECS  software  maintenance  profile.  This  profile  specifies 
the  expected  maintenance  actions  which  are  required  in  order  to  support  » 
the  mission  objectives  of  the  operational  system. 

An  example  profile  might  include: 

a)  Types  of  priority  support  requests  (e.g.,  emergency, 
urgent,  normal) 

b)  Expected  response  time  for  support  requests  (e.g., 

24  hours  for  emergency,  1  week  for  urgent,  1  month  for 

normal ) 

c)  Expected  number  (perhaps  even  distribution)  of  support 

requests  of  each  priority  type  over  a  specified  (e.g., 

1  year)  period  of  time  (e.g.,  10  emergency,  20  urgent, 

100  normal ) 

d)  Expected  number  of  support  requests  by  the  above  cate¬ 
gories  and  by  type  of  maintenance  action  (correction, 
enhancement,  conversion) 

e)  Because  support  requests  of  a  given  type  can  vary  greatly 
in  complexity,  it  may  be  desirable  to  specify  the  above 
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information  further  categorized  (low,  med,  high)  by  com¬ 
plexity  with  an  example  of  each  complexity  category  for 
■guidance. 

The  support  profile  is  defined  at  least  for  each  ECS  and  represents 
a  management  tracking  history  of  ECS  support  activity.  It  may  also 
happen  that  a  given  set  of  resources  (e.g.,  personnel,  support  systems, 
facilities)  supports  more  than  one  ECS.  In  this  case  there  will  need  to 
be  a  way  to  link  the  two  (or  more)  profiles  so  as  to  reflect  the  risk  due 
to  this  overlapping  responsi bi 1 ity.  For  example,  it  may  be  very  possible 
for  resources  to  support  an  emergency  request  for  either  of  two  ECSs,  but 
not  both  at  the  same  time. 

The  support  profile  is  a  top  level  support  activity  requirements 
specification  between  any  two  of  the  risk  agents.  For  example,  a  devel¬ 
oper  may  be  designing  software  products  so  that  a  particular  support  pro¬ 
file  can  be  met.  Upon  evaluation  (e.g..  an  OT&E  software  supportabi  1  i ty 
test)  the  supporter  may  determine  that  the  "agreed  upon"  support  profile 
is  violated  in  some  way.  For  example,  a  low  complexity  emergency  soft¬ 
ware  correction  may  take  40  hours  on  the  average  to  identify,  correct, 
configure,  and  distribute,  whereas  the  agreed  upon  profile  requirement  is 
24  hours.  At  this  point  the  supporter  has  experienced  a  potential 
negative  outcome.  How  negative  the  outcome  is,  that  is,  the  degree  of 
risk,  may  depend  upon  other  risk  agents  as  well,  such  as  the  user.  If 
the  user  can  agree  that  40  hours  is  acceptable,  then  there  is  no  negative 
outcome.  On  the  other  hand,  the  supporter  may  have  been  less  experience 
with  the  system  than  the  developer  expected,  or  perhaps  not  all  the 
necessary  support  tools  such  as  cross  reference  maps  were  available 
during  the  test.  In  this  case,  the  alternative  may  be  to  reevaluate  the 
test  results  under  different  environmental  conditions.  All  of  these 
alternative  choices  and  ways  to  reduce  the  potential  risk  are  considered 
as  part  of  the  risk  analysis  process. 


1 1 1-9 


Jimm  iwlwwui  uiuilhuiui  uim  w  wm  PJ>U»JJ  m  m 

i 

i 
* 

|  THE  BDM  CORPORATION  BDM/A-84-496-TR 


3.3. 1.3  Estimating  Probability  of  Negative  Outcome  Occurrences 
and  Magnitude  of  Consequence  Value-! 


The  major  problems  are  in  estimating  what  the  magnitude  of  the 

consequence  will  be  when  a  deviation  from  the  required  support  profile 
(negative  outcome)  occurs,  and  what  the  probability  is  that  any  given 
deviation  will  occur.  Once  this  process  is  complete  for  all  applicable 
potential  negative  outcomes,  a  software  supportability  risk  baseline  has 
been  established.  Techniques  for  estimating  these  probabilities  and 
magnitudes  are  discussed  in  section  IV  of  this  document. 

A  major  aspect  of  using  the  above  described  risk  baseline  is  to 
determine  the  relationship  between  the  measurable  software  supportability 
factor  characteristics  (see  section  3.3.2)  and  the  support  profile.  For 
example,  the  "quality"  of  the  source  code  (as  evidenced  by  the  source 
code  listings)  will  have  an  effect  upon  how  rapidly  and  effectively  soft¬ 
ware  maintenance  actions  can  be  accomplished.  How  much  effect  is  not 

known.  Currently  AFOTEC  measures  the  characteristics  of  software  main¬ 
tainability  on  a  relative  scale  of  1  (worst)  to  6  (best).  A  score  of  2.0 
may  result  in  the  inability  of  support  resources  to  satisfy  a  required 
software  support  profile  characteristic.  However,  it  will  clearly  depend 
upon  what  the  profile  requirement  is  and  what  the  support  environment 
(SSF)  capabilities  are.  It  is  useful  to  have  some  way,  preferably  a 

mathematical  algorithm,  of  defining  what  the  relationships  among  these 
factors  are. 

An  extended  example  using  current  AFOTEC  software  supportability 

factors  will  illustrate  some  of  the  concepts  of  estimating  risk.  Suppose 
for  a  given  software  support  facility  (evaluation  score  of  3.3),  and  the 
associated  ECS  software  (evaluation  score  of  3.0),  and  the  required  soft¬ 
ware  supportabil ity  risk  profile  baseline  (with  the  requirement  for 
24  hour  turnaround  for  low  complexity  emergency  maintenance  correction 
requests),  it  is  estimated  that  the  probability  is  0.4  that  the  negative 
outcome  of  25  hour  (1  hour  delay)  turnaround  will  occur.  The  magnitude 
of  the  consequence  might  be  estimated  as  a  minor  inconvenience  to  opera¬ 
tional  readiness  (during  peacetime).  This  describes  one  point  on  a 
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family  of  related  curves  called  probability  density  functions  (PDFs).  As 
each  factor  variable  is  allowed  to  range  over  possible  values,  or  the 
potential  negative  outcome  is  allowed  to  vary,  PDFs  are  derived.  Such  a 
probability  density  function  (PDF)  curve  is  created  as  in  figure  3-1. 
This  PDF  curve  is,  of  course,  hypothesized.  From  this  PDF,  the  accum¬ 
ulated  risk  across  a  range  of  possible  outcomes  can  be  determined  simply 
by  determining  the  area  under  that  part  of  the  curve  (i.e.,  integrating 
over  the  PDF  from  the  lower  range  value  to  the  upper  range  value).  More 
specific  information  on  the  theoretical  foundations  of  risk  is  discussed 
in  section  4.2  of  this  report. 

As  is  easily  noted,  this  process  could  become  complex,  especially  if 
the  number  of  variables  is  large  or  the  data  upon  which  probability  esti¬ 
mation  is  to  be  done  is  not  available.  Fortunately,  precise  numeric 
values  as  used  in  the  illustration  above  are  not  always  needed  to  obtain 
a  feel  for  the  risk  (e.g.,  low,  medium,  high).  Techniques  as  described 
in  section  4.3  of  this  report  are  applicable  for  both  subjective  and 
objective-derived  data. 


3.3. 1.4  Reducing  Risk  and  Choosing  Alternatives. 


The  process  of  reducing  risk  and/or  choosing  alternatives  is  part  of 
a  general  risk  aversion  process.  Knowing  the  potential  effect  due  to  a 
negative  outcome  will  cause  aversive  action  to  be  taken.  Returning  to 
the  example  in  section  3. 3. 1.3  of  the  software  supportabi 1 i ty  evaluation, 
one  way  to  reduce  risk  is  to  require  a  developer  to  obtain  a  score  of 
more  than  3.0  as  a  "preventive"  measure.  Another  possibility  is  to 
require  modifications  to  the  source  code  to  correct  the  major  identified 
deficiencies  so  that  an  overall  score  of  3.0  or  better  will  be  attained 
prior  to  acceptance. 

Other  alternatives  can  include  upgrading  related  variables  (e.g., 
the  SSF)  sufficient  to  decrease  the  risk.  Or,  perhaps  the  original 
requirement  of  24  hour  turnaround  on  emergency  requests  for  low  com¬ 
plexity  corrective  maintenance  can  be  relaxed.  The  ultimate  alternatives 
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•are  to  do  nothing,  or  to  cancel  any  further  activity  on  the  particular 
subject  software. 

A  key  part  of  the  analysis  to  reduce  risk  and/or  choose  alternatives 
is  the  predetermination  of  the  cost  and  benefit  of  each  possible  course 
of  action.  This  should  be  carefully  (and  probably  simultaneously)  inte¬ 
grated  with  the  acceptability  of  the  identified  risk  to  the  risk 
agent (s ) . 


3. 3. 1.5  Acceptance  of  Risk. 

A  risk  agent  may  require  an  analysis  of  risk  reduction  and  alterna¬ 
tive  choices  before  acceptability  can  be  determined.  Or,  a  risk  agent 
may  be  able  to  directly  specify  acceptance  of  the  risk.  Once  risk  has 
been  identified  and  the  magnitude  of  potential  negative  outcomes  deter¬ 
mined,  it  is  recommended  that  the  risk  agent  be  consulted  as  to  the 
acceptability  of  the  risk  prior  to  further  analysis.  The  analysis 
process  itself  may  be  costly  and  the  potential  for  risk  reduction  small. 

Risk  acceptance  has  very  broad  implications.  In  one  case  an 
individual  user  may  accept  support  risk  on  a  particular  software  package. 
In  another  case  the  risk  acceptance  may  involve  a  group  of  people  as  the 
risk  agent  and  a  degradation  in  national  defense  due  to  reduced  opera¬ 
tional  effectiveness  (availability)  of  a  fighter  aircraft.  Risk  accept¬ 
ance  may  involve  regulatory  agencies,  societal  standards  and  concerns, 
and  other  political  influences.  Whenever  there  is  risk  it  is  likely  that 
there  will  be  more  than  one  risk  agent,  but  there  may  be  a  large  dis¬ 
parity  between  the  degree  of  risk  which  must  be  assumed  by  any  one  agent. 
This  creates  an  atmosphere  of  "unfairness"  which  may  be  on  the  fringe  of 
psychological  perception.  In  any  case,  balancing  risk  among  risk  agents 
is  a  part  of  the  process  to  reduce  risk  and  choose  alternatives  and  may 
be  required  as  a  prerequisite  to  risk  acceptance. 

Returning  to  the  example  in  section  3. 3. 1.3,  suppose  the  following 
consequences  were  based  upon  the  possible  negative  outcome  of  an 
excessive  (more  than  24  hour)  time  to  correct  an  emergency  low  complex 
software  fault: 
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a)  0-1  hours  -  Minor  inconvenience 

b)  1-2  hours  -  Readiness  affected 

c)  2-3  hours  -  Readiness  impaired 

d)  >3  hours  -  Emergency  Consequence 

The  level  of  risk  acceptance  depends  on  the  probability  of  occurrence  and 
the  nature  and  magnitude  of  the  value  of  the  consequences  that  can  occur. 
This  is  qualitatively  illustrated  in  figure  3-2  for  a  single  risk  agent 
for  our  example.  The  abscissa  shows  an  evenly  spaced  rank  scale  of 
consequence  value  in  terms  of  the  gross  indication  of  the  hierarchy  of 
risk  consequences  arbitrarily  assigned  above.  A  logarithmic  scale  of 
probability  of  occurrence  appears  on  the  ordinate.  Changes  in  the 
spacing  in  the  abscissa  scale  will  alter  the  specific  shape  of  the  curve, 
but  not  the  general  downward  slope  to  the  right  hand.  In  other  words, 
consequences  not  involving  emergency  consequences  have  higher  acceptable 
levels  of  probability. 

The  acceptable  probability  of  occurrence  of  a  specific  consequence 
value  is  designated  as  a  "risk  acceptance  level".  The  profile  of  the 
acceptability  of  the  probability  of  occurrence  for  all  consequences 
involved  in  a  situation  is  designated  a  "risk  acceptance  utility  func¬ 
tion". 

Figure  3-2  illustrates  two  alternate  risk  acceptance  utility  func¬ 
tions.  The  top  curve  represents  the  risk  acceptance  utility  function  for 
a  risk  taker  with  a  lower  "propensity  for  risk  acceptance"  than  the 
original  risk  agent.  The  bottom  curve  illustrates  a  higher  propensity 
for  risk  acceptance;  that  is,  the  risk  taker  is  more  likely  to  "take  a 
chance"  than  the  original  risk  agent. 

More  details  concerning  risk  acceptance  and  estimating  risk  accep¬ 
tance  can  be  found  in  Rowe's  book  (reference  5.25)  and  several  other 
references.  Some  of  the  estimating  techniques  discussed  in  section  IV  of 
this  report  are  applicable. 


3. 3. 1.6  Uncertainty. 


With  every  descriptive  process  there  is  uncertainty.  With  every 
measurement  process  there  is  uncertainty.  In  order  for  AFOTEC  to  develop 
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an  RAMSS.  it  will  be  necessary  to  understand  and  control  the  uncertainty 
in  the  risk  assessment  process. 

It  is  not  possible  to  completely  identify  all  the  software  support- 
ability  factors.  Thus,  a  model  is  created  as  an  approximation  to  the 
reality.  Currently,  this  model  is  a  hierarchy  of  increasingly  more 
specific  characteristics  of  software  supportabi 1 ity  (see  section  3.3.2 
and  reference  5.1  for  more  details).  This  disparity  between  the  "real" 
description  and  the  "model"  description  of  software  supportabi 1 i ty  is 
termed  descriptive  uncertainty. 

With  each  characteristic  in  the  descriptive  model  hierarchy  there  is 
a  corresponding  evaluation  measurement  scale.  Scales  for  measurement 
include: 

a)  Nominal  Scale  ( Identify-Taxonomy) .  A  classification  of 
items  that  can  be  distinguished  from  one  another  by  one  or 
more  properties. 

b)  Ordinal  Scale  (Order-Rank).  An  ordering  (ranking)  of, 
items  by  the  degree  they  obtain  some  criterion. 

c)  Cardinal  Scale  (Interval).  A  continuous  scale  between  two 
end  points,  neither  of  which  is  necessarily  fixed. 

d)  Ratio  Scale  (Zero  reference).  A  cardinal  scale  with  one 
end  point  fixed  by  reference  to  an  absolute  physical  end 
point  (e.g.,  absolute  zero  on  the  Kelvin  temperature 
scale),  from  which  are  developed  other  cardinal  scales, 
all  of  which  are  related  by  simple  ratios. 

The  proposed  Fisk/Murch  model  (reference  5.12)  is  based  upon  a  six 
point  scale  of  measurement:  F  -completely  disagree;  E;  0;  C;  3;  A  - 
completely  agree,  which  can  also  be  considered  to  have  integer  values 
from  1  to  6.  This  scale  is  a  cardinal  scale.  There  is  uncertainty  in 
the  scale  and  with  the  scale  values  assigned  to  software  supportabi  1  i ty 
characteristics  as  part  of  an  evaluation.  This  is  called  measurement 
uncertainty.  As  values  with  measurement  uncertainty  are  aggregated  to 
obtain  "higher  level"  information,  the  measurement  uncertainty  is  also 
aggregated.  This  is  the  "compound  error"  problem. 
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One  of  the  major  requirements  of  an  RAMSS  for  AFOTEC  is  to  properly 
balance  the  descriptive  uncertainty  due  to  lack  of  model  fidelity  with 
the  measurement  uncertainty  of  evaluating  model  characteristics, 
estimating  potential  for  negative  software  supportabi 1 ity  outcomes,  and 
assessing  the  magnitude  of  the  consequence  and  acceptability  level  to  the 
user  and  supporter  of  the  software. 

3.3.2  Evaluation  Model  Framework. 


An  RAMSS  is  by  definition  a  model  of  real  software  supportabi 1 ity 
risk.  The  fidelity  of  such  a  model  will  depend  upon  the  defined  evalua¬ 
tion  factors  of  software  supportabi 1 ity,  factor  criteria  and  lower  level 
characteristics,  and  the  capability  of  the  model  to  relate  measures  of 
these  characteristics  to  software  supportabi 1 ity  risk  baselines.  An 
evaluation  model  framework  which  currently  follows  AFOTEC  implemented 
evaluation  methodologies  is  shown  in  figure  3-3.  This  figure  illustrates 
the  modeling  of  software  supportabil ity  through  a  hierarchy  of  factors, 
criteria,  and  characteristics  in  succeedingly  more  detailed  descriptive 
levels.  At  each  level,  a  measurement  scale  is  applied  to  the  evaluation 
of  the  information  at  that  level.  Note  that  as  the  descriptive  certainty 
of  the  information  increases  (i.e.,  one  proceeds  to  lower  levels  of 
detail  in  the  hierarchy),  the  uncertainty  in  accummul ated  measurement 
accuracy  (combination  of  lower  level  measures  into  higher  level  measures) 
also  increases.  These  inversely  desirable  results  mean  that  the  risk 
assessment  level,  i.e.,  the  lowest  hierarchy  level  on  which  risk  assess¬ 
ment  is  based,  must  be  carefully  determined  so  that  overall  uncertainty 
in  the  assessment  results  is  minimized. 

The  current  AFOTEC  software  supportabi 1 ity  factors  plus  some  recom¬ 
mended  new  factors  (dotted  lines)  are  shown  in  figure  3-4.  All  elements 
of  figure  3-4  except  those  in  dotted  boxes  are  explained  in  detail  in 
references  5.1  and  5.12.  Only  the  elements  in  dotted  boxes  will  be 
briefly  described. 

The  software  support  management  evaluation  is  necessary  to  allow  for 
variance  in  software  support  risk  which  may  be  more  directly  a  management 
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function  than  can  be  identified  by  either  software  product  quality  or  the 
software  support  capabilities.  This  evaluation  has  at  least  two  test 
factors:  design  and  configuration  management. 

One  of  the  important  links  to  software  supportabi lity  OT&E  is  the 
software  system  design  which  is  currently  only  evaluated  in  a  limited  way 
(and  then  only  through  formal  specifications).  This  design  for  software 
supportabi lity  is  captured  in  the  software  products,  the  Operational/ 
Support  Configuration  Management  Procedures,  the  various  test  plans,  the 
Computer  Resources  Integrated  Support  Plan,  and  by  various  development 
management  techniques  (e.g.,  chief  programmer  team,  rapid  prototyping) 
and  acquisition  management  techniques  (e.g.,  risk  monitoring  through  Data 
Item  Description  metric  requirements ,  I V&V  risk  assessment).  Design  of  a 
system  for  the  mission  objective  and  software  supportabi 1 ity  is  critical 
to  reducing  support  risks  (e.g.,  excessive  support  costs,  degraded  turn¬ 
around)  . 

An  OT&E  software  supportabi lity  design  evaluation  can  be  done  early 
in  the  full  scale  development  (or  even  demonstration  and  validation) 
acquisition  phase.  It  would  provide  early  guidance  to  AFOTEC  for 
adopting  a  test  and  evaluation  strategy  during  OT&E.  This  strategy  could 
thus  stress  identified  risk  drivers  (such  as  particular  functional  sub¬ 
systems  or  modules)  and  help  optimize  the  application  of  limited  evalua¬ 
tion  resources  to  maximize  risk  identification.  This  would  lead  to  a 
reduced  uncertainty  in  the  residual  risk  eventually  identified  through 
the  adopted  test  strategy  during  OT&E  at  the  production  and  deployment 
acceptance  decision  point. 

There  are  two  major  processes  which  characterize  the  software 
support  function.  One  is  the  software  maintenance  process.  The  other 
process  is  software  configuration  management.  The  essence  of  most  of  the 
AFOTEC  SSF  evaluation  is  in  measuring  the  capability  of  the  SSF  to 
support  the  software  maintenance  process  within  the  bounds  of  the  ECS 
maintenance  profile. 

Part  of  the  SSF  evaluation  capability  is  to  specify  an  "other 
support  system"  (reference  figure  3-4)  as  a  specific  configuration 
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management  system.  Some  of  the  technical  characteristics  of  configura¬ 
tion  management  can  be  evaluated  in  this  manner  to  minimal  depth. 
However,  a  major  part  of  configuration  management  j_s  management,  i.e., 
procedures,  administration,  control  boards,  and  organizational  inter¬ 
faces.  The  AFOTEC  SSF  evaluation  does  not  capture  these  management 
aspects.  This  "structure"  begins  to  form  very  early  in  the  sytsem 
acquisition  life  cycle.  Frequently  project  management  places  emphasis 
upon  development  configuration  management  with  little  attention  paid  to 
the  transition  from  development  to  support  or  the  support  configuration 
management.  OT&E  needs  to  be  aware  of  early  acquisition  decisions  and 
risk  evaluations  concerning  the  software  configuration  management  plan 
and  how  that  plan  specifies  the  transition  and  follow  on  support 
concepts.  The  Computer  Resources  Integrated  Support  Plan  and  the  Opera¬ 
tional/Support  Configuration  Management  Procedures  should  highlight  major 
aspects  of  what  those  requirements  need  to  be. 


3.3.3  Elements  of  Risk  Management  Model. 


A  software  supportabi 1 ity  (SS)  risk  management  model  incorporates: 
the  software  supportabi 1 ity  test  and  evaluation;  the  risk 
assessment/analysis  framework  as  applied  to  software  supportabi 1 ity 
concerns  (the  RAMSS);  the  software  supportabi 1 ity  risk  management 
function  conducted  by  support  MAJCOMs  or  Special  Operating  Agencies;  and 
the  appropriate  system  risk  management  by  organizational  agencies  (e.g., 
the  Designated  Approving  Authority  and  other  agencies  are  interested  in 
risk  assessment  for  concerns  in  addition  to  software  susceptibility). 
The  basic  elements  of  such  a  model  are  illustrated  in  figure  3-5. 

The  model  illustrates  the  interrelationships  among  management  (e.g., 
the  Designated  Approving  Authority  at  the  mission  system  management 
level,  and  program  support  management  at  the  ECS  level),  the  technical 
aspects  of  measuring  software  supportabi 1 ity  through  test  and  evaluation, 
and  the  analysis  of  potential  negative  and  positive  outcomes  for  software 
supportabi 1 ity  impact  and  risk.  A  generalized  process  flow  which  repre¬ 
sents  a  typical  sequence  of  risk  management  activity  within  the  model 
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Figure  3-5.  SS  Risk  Management  Model  Framework 


1 11-22 


THE  BDM  CORPORATION 


BDM/A-S4-496  TR 


> 

i 

I 

i 


framework  is  shown  in  figure  3-6.  The  following  paragraphs  will  discuss 
figure  3-5.  the  model  framework,  from  bottom  to  top. 


3. 3. 3.1  System  Management. 

System  management  controls  the  overall  mission  objectives  and  the 
allocation  of  those  objectives  to  system  requirements .  The  results  of 
system  acceptance  tests  are  reviewed  against  mission  objectives  prior  to 
system  approval.  This  function  is  primarily  allocated  to  the  system 
Designated  Approving  Authority,  assisted  by  appropriate  Air  corce 
agencies,  MAJCOMS,  Special  Operating  Agencies,  and  so  forth,  as 
necessary. 


3. 3. 3. 2  SS  Risk  Management. 

SS  Risk  Management  includes  coordination  of  risk  analysis  functions, 
integration  with  SS  OT&E.  and  approval  throughout  the  system  life  cycle. 
Acceptance  is  the  result  of  software  specification  requirements  being  met 
through  technical  SS  test  and  evaluations.  SS  risk  management  is 
responsible  for  determining  whether  the  uncertainty  in  these  results  as 
described  through  the  risk  analysis  process  is  acceptable  for  formal 
acceptance  test  requirements.  The  basic  risk  parameters  as  described  in 
terms  of  cost,  schedule,  and  technology  impact  are  major  inputs  to  this 
decision  process.  SS  risk  management  also  passes  the  overall  software 
system  objectives  to  the  risk  analysis  process  for  consideration.  And, 
SS  risk  management  presents  the  results  of  any  risk  analysis  as  necessary 
to  system  management  prior  to  any  final  approval  decisions. 

3. 3. 3. 3  SS  Risk  Analysis. 

The  SS  Risk  Analysis  framework  is  based  upon  risk  determination: 
the  process  of  identifying  and  estimating  the  magnitude  of  risk;  and, 
risk  evaluation:  the  complex  process  of  determining  acceptable  levels  of 
risk  and  alternative  risk  choices. 
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a)  Identification.  Identification  includes  determination  of 
the  risk  agents,  risk  analysis  team  members,  SS  assets, 
overall  SS  objectives,  potential  negative  events,  possible 
SS  MOEs,  required  schedules  and  deliverables,  and  the 
system  sensitivity  and  process  criticality.  Identifica¬ 
tion  also  includes  recognition  of  any  changes  or  new 
relationships  in  any  of  the  above  risk  parameters. 

b)  Estimation .  Estimation  involves  determination  (either 
quantitatively  or  qualitatively)  of  the  likelihood  of  an 
identified  risk  occurring,  the  frequency  of  occurrence, 
and  the  criticality  or  consequence  if  it  does  occur. 
These  estimates  are  dependent  upon  the  risk  agent  (e.g., 
DAA,  MAJCOM/SOA,  OPR,  developer,  using/supporting  command) 
and  are  represented  on  scales  commensurate  with  the 
required  risk  analysis  detail  as  identified  in  a). 

c)  Aversion.  Aversion  involves  the  evaluation  of  how  the 
estimated  level  of  risk  might  be  reduced  through  alterna¬ 
tives  and  avoidance,  and  by  what  degree.  Again,  the  scale 
of  measurement  might  be  (low,  medium,  high)  or  a  detailed 
probability.  Residual  risk  is  determined  after  all 
alternatives  and  avoidance  techniques  have  reduced  the 
risk  as  much  as  possible. 

d)  Acceptance .  Acceptance  represents  the  evaluated  willing¬ 
ness  of  the  particular  risk  agent  to  accept  a  specific 
level  of  risk  (the  residual  risk)  to  obtain  some  gain  or 
benefit.  Part  of  this  process  is  to  properly  represent 
all  risk  decision  parameters  so  that  acceptance  can  be 
reasonably  ascertained  directly  from  the  parameters. 

Risk  estimation  is  an  essential  part  of  both  the  estimation  and 
aversion  functions.  Potential  qualitative,  quantitative,  and  hybrid 
techniques  which  could  be  used  are  described  in  section  IV  of  this  docu¬ 
ment.  Economic  assessment  is  part  of  both  the  aversion  and  acceptance 
functions.  The  economic  assessment  provides  the  means  for  determining 
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the  feasibility  and  relative  value  of  the  alternate  software  support- 
ability  risk  reduction  measures.  A  somewhat  broader  interpretation  of 
"economic"  to  mean  not  just  a  dollar  cost,  but  time,  labor  and  perhaps 
other  resource  cost  is  very  useful  since  it  may  not  be  possible  to  reduce 
a  software  supportabi 1 ity  residual  risk  to  a  dollar  cost.  More  detailed 
information  concerning  the  risk  analysis  aspects  of  this  model  can  be 
found  in  reference  5.25. 

3. 3. 3. 4  $5  OT&E . 

The  elements  of  an  SS  OT&E  are  briefly  described  in  section  3.3.7. 
The  SS  OT&E  process  is  described  in  reference  5.1.  The  importance  of 
SS  OT&E  to  risk  management  is  in  the  derivation  of  SS  evaluation  measures 
for  use  in  risk  analysis.  In  addition,  there  is  significant  interdepen¬ 
dence  among  SS  OT&E;  risk  identification  of  potential  negative  events  and 
SS  risk  reduction  measures;  and  risk  aversion  analysis  of  alternatives 
and  degree  of  risk  reduction.  During  much  of  the  process,  the  distinc¬ 
tion  between  risk  analysis  and  SS  OT&E  may  be  imperceptible,  but  the  two 
areas  do  have  reasonably  distinct  general  objectives. 

3. 3. 3. 5  Model  Characteri sties . 

At  any  level  of  the  model,  iteration  with  adjacent  levels  is  very 
likely.  For  example,  identification  of  SS  assets,  potential  exposure  to 
SS  risks,  SS  measures,  and  so  forth  derives  partially  from  system  mission 
objectives  and  partially  from  a  technology  assessment  of  SS  OT&E  results. 
Once  the  risk  identification  function  has  been  "completed",  SS  OT&E  will 
determine  values  of  SS  measures  which  can  be  used  to  integrate  into  the 
risk  estimation  and  aversion  process.  But,  this  process  may  uncover 
other  potential  risk  exposures  previously  identified,  but  which  should  be 
considered.  Thus,  the  risk  identification  function  is  "reopened"  for 
consideration.  In  a  like  manner,  most  of  the  other  elements  may 
iteratively  affect  nearly  any  other  element.  It  may  happen  that  a  par¬ 
ticular  SS  OT&E  result  has  such  a  large  system  impact  that  all  "inter¬ 
mediate"  levels  are  "skipped"  to  determine  possible  alternative  mission 

r  n-26 


THE  BOM  CORPORATION 


BDM/A-84-496  TR 


objectives.  Of  course  the  intermediate  levels  are  not  really  skipped. 
Instead,  the  levels  become  merged  over  a  short  time  period  (this  is 
popularly  called  "crisis  management"  or  "putting  out  the  fires").  This 
is  done  in  order  to  make  timely  decisions  to  solve  a  problem  which  could 
have  critical  impact  (cost,  schedule,  or  technology)  upon  the  complete 
system. 

3.4  MEASURES  OF  SOFTWARE  SUPPORTABILITY  RISK. 

There  are  several  measures  (or  metrics)  which  are  integral  to  soft¬ 
ware  supportabi 1 ity  risk  assessment  process. 

a)  ECS  SS  Profile  Metrics 

b)  SS  Evaluation  Metrics 

c)  SS  Negative  Outcome  Probability  Estimates 

d)  SS  Magnitude  of  Consequence  Estimates 

e)  SS  Risk  Levels 

f)  Risk  Agent  Acceptance  Levels 

Figure  3-7  integrates  these  measures  loosely  with  the  risk  assess¬ 
ment  process  to  illustrate  the  risk  measure  derivation.  The  OT&E  objec¬ 
tives  and  MOEs  provide  the  force;  the  baseline  support  requirements  and 
support  evaluation  metrics  are  the  anchor  in  the  process;  the  estimation 
metrics  are  derived  from  heuristics,  experimental  tests,  or  historical 
data  using  techniques  in  section  IV;  risk  levels/metrics  are  simply 
integration  of  estimated  negative  outcome  probability  density  functions 
over  a  specified  range  of  outcomes;  and  the  risk  levels  plus  magnitude  of 
consequence  metrics  are  considered  by  the  risk  agent  in  determining  the 
risk  acceptance  levels. 

3.4.1  ECS  SS  Profile  Requirements  Metrics. 

The  following  categories  create  a  maximum  of  27  requirements: 

a)  Priority  Type:  E  -  Emergency 

U  -  Urgent 
N  -  Normal 
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b)  Maintenance  Action  Type: 


c)  Complexity  Level : 


C  Correction 
A  -  Enhancement 
V  -  Conversion 
L  -  Low 
M  -  Medium 


H  -  High 

For  example,  the  requirement  for  software  support  turnaround  for  an 
emergency  low  complexity  corrective  maintenance  action  (ELC)  might  be  one 
hour.  Other  requirement  metrics  are  not  excluded  from  consideration  in 
this  profile. 

3.4  2  SS  Evaluation  Metrics. 

The  SS  Evaluation  Metrics  include  the  current  AFOTEC  software 
support  facility  and  software  maintainability  metrics  as  well  as  any  new 
metrics  developed  in  the  future.  A  new  set  of  software  support  manage¬ 
ment  metrics  for  design  and  configuration  management  were  suggested  in 
section  3.3.2.  The  AFOTEC  SS  evaluation  measures  are  described  in 
reference  5.1. 

3.4.3  SS  Negative  Outcome  Estimates. 

Using  techniques  defined  in  section  IV,  an  estimate  of  likelihood  of 
occurrence  for  each  negative  outcome  to  be  considered  is  made.  Only  the 
negative  outcomes  which  might  be  expected  to  drive  the  OT&E  software 
supportabil ity  risk  acceptance  should  be  considered.  A  family  of  proba¬ 
bility  density  functions  F^  =  (P.j  :  i=l..n}  (which  may  well  be  "standard" 
density  functions  as  initial  estimates)  is  derived  for  each  class  (k)  of 
negative  outcomes.  Such  a  class  might  consist  of  all  the  Pi  as  the 
software  maintainability  evaluation  metric  is  allowed  to  vary  from  1  to  6 
in  discrete  steps  (with  the  set  of  fixed  variables  as  in  the  example  of 
section  3.3.1  3) . 
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The  use  of  classes  and  families  of  probability  density  functions  is 
necessary  only  for  sensitivity  analysis  for  risk  reduction  and  alterna¬ 
tive  choices  and  may  only  consist  of  two  or  three  members.  The  family  of 
probability  density  functions  for  the  example  in  section  3. 3. 1.3  are  of  a 
Rayleigh  curve  form.  Once  the  parametri zation  for  the  curve  is  deter¬ 
mined,  the  specific  family  member  shape  is  determined  and  the  derivations 
of  the  estimation  metrics  is  complete. 


3.4.4  SS  Magnitude  of  Consequence  Estimates. 


These  metrics  are  a  function  of  the  set  of  negative  outcomes,  the 
probability  of  negative  outcome  occurrence,  and  the  risk  agent's  assess¬ 
ment.  This  latter  assessment  may  be  objective  or  subjective.  Objective 
assessments  are  measured  behavioral  responses  of  the  risk  agent  to  the 
negative  outcome  consequence.  Normally,  this  objective  measurement  is 
not.  possible  for  software  supportability  negative  outcomes.  One  possible 
example  of  this  would  be  measuring  the  reaction  of  a  user  to  varying 
interactive  response  times  because  of  software  modifications.  Subjective 
assessments  by  the  risk  agent  based  upon  subjectively  or  objectively 
derived  data  (e.g.,  history  of  similar  system)  is  more  likely  to  occur. 

The  metrics  may  be  represented  on  any  of  the  possible  scales  such  as 
(10,  MED,  HI),  (INCONVENIENT,  DANGEROUS,  CATASTROPHIC),  and  so  forth. 


3.4.5  SS  Risk  Levels. 


Integration  (or  summation  if  the  probability  density  function  is 
discrete)  of  the  probability  density  function  over  a  range  of  possible 
outcomes  results  in  a  specific  software  supportability  risk  level.  In 
the  example  of  section  3. 3. 1.3,  if  it  is  desired  to  determine  the  risk 
due  to  negative  outcomes  of  greater  than  or  equal  to  3  hours  and  p  is  the 
probability  density  function,  then  the  risk  R  is  given  by: 


/  Pd* 
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Assuming  the  density  function  is  a  Rayleigh  curve  (which  it  appears  to 
resemble),  we  might  obtain: 

00  w  2 

R  *  /  a  e'bx  dx 
3 

where  "a“  and  “b"  are  parameters  which  determine  the  curve  shape. 

This  same  technique  can  be  used  to  determine  risk  level  for  the 
discrete  case,  even  for  use  of  scales  such  as  (LO,  MED,  HI).  The  usual 
technique  is  to  assign  a  nuneric  value  on  an  even  interval  scale  to  eacn 
of  the  fuzzy  linguistic  terms  and  then  use  normal  summation  indexing. 

3.4.6  Risk  Agent  Acceptance  Levels. 

The  acceptance  level  is  simply  that  particular  value  of  the  risk 
level  which  is  acceptable  to  a  specific  risk  agent.  Implicit  in  this 
acceptance  is  the  acceptance  of  the  probability  of  the  negative  outcome. 

For  the  example  of  section  3. 3. 1.3,  the  probability  of  0.4  that  an 
emergency  low  complexity  correction  maintenance  action  will  take  one  hour 
more  than  the  required  24  may  not  be  acceptable  to  the  user  risk  agent. 
Likewise,  by  altering  the  curve  shape  parameter  so  that  a  user  acceptable 
value  of  0.3  is  attained  might  be  unacceptable  to  the  supporter  risk 
agent  based  upon  the  SSF  and  SWM  evaluation  measures.  However,  if  the 
supporter  can  obtain  one  more  person  with  systems  analysis  background, 
perhaps  the  SSF  evaluation  metric  will  be  improved  enough  so  that  the 
latter  probability  value  of  0.3  is  also  acceptable  to  the  supporter  risk 
agent. 

Various  techniques  for  reduction  of  risk  or  selection  of  alternative 
courses  of  action  may  be  necessary  before  the  iteration  of  all  risk 
agents  involved  to  acceptable  levels  of  risk  is  attained. 

3.5  REPORTING  SOFTWARE  SUPPORTABILITY  RISK. 


AFOTEC  has  a  well-defined  process  for  reporting  results  of  an  0T4E 
evaluation  (reference  5.40).  Figure  3-8  summarizes  the  possible  types  of 
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reports  produced  as  part  of  OT&E.  Since  there  are  strict  limits  in  prep¬ 
aration  time,  report  length,  and  report  content  which  are  specified,  any 
SS  risk  assessment  results  must  be  integrated  into  this  more  general  OT&E 
report  process. 

The  technical  details  covered  throughout  most  of  this  report  are 
inappropriate  to  include  in  most  of  the  indicated  reports  except  in 
excerpt  form.  Most  of  the  technical  details  such  as  distribution  func¬ 
tions,  specific  test  factor  component  and  component  evaluation  metrics, 
probability  estimation  techniques,  and  so  forth,  should  be  available  as 
backup  support.  In  case  the  summary  presentation  forms  create  concerns 
which  require  further  explanation,  the  detailed  information  in  computer 
or  typed  report  form  can  be  referenced. 

The  summary  presentation  form  can  take  the  form  of  graphs,  tables, 
or  matrices  appropriately  highlighted  during  briefings  to  show  points  of 
interest.  The  user/supporter  risk  matrix  suggested  in  reference  5.12  is 
an  excellent  example  of  a  risk  report.  Reference  5.1  and  5.41  contain 
tables  which  summarize  the  reporting  of  software  maintenance  evaluation 
metrics.  Essentially,  each  of  the  metrics  described  in  section  3.4 
should  be  reported  in  a  storyboard  fashion.  This  technique  would  act  as 
a  summary  of  the  risk  assessment  process,  caveats  and  assumptions  could 
be  inserted  as  appropriate,  and  the  key  results  such  as  evaluation 
metrics,  risk  levels,  risk  reduction  and  alternate  choice  analysis,  and 
final  acceptance  levels  with  risk  residues  could  be  presented. 

Such  summary  information  could  be  easily  included  to  varying  levels 
of  depth  in  each  of  the  report  types  indicated  in  figure  3-8. 


3.6  LEVEL  OF  EFFORT  TO  DEVELOP  AND  IMPLEMENT  A  CANDIDATE  RAMSS. 


Oespite  the  fact  that  a  methodology  or  technique  has  not  been 
selected,  it  is  possible  to  take  a  preliminary  look  at  the  level  of 
effort  required  to  develop  and  implement  an  RAMSS  and  the  phasing  of  such 
an  effort.  Table  3-3  shows  the  anticipated  phases,  resources,  and 
results  for  each  phase. 
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Table  3-3. 

RAMSS  Phased  Development 


Phase 

Function 

Resources 

Results 

I 

Concept 

(Current  Phase) 

5  Months 

3.5  People 

RAMSS  Framework 

RAMSS  Potential  Techniques 
RAMSS  Integration  with 

CSS  OT&E 

Possible  RAMSS  Measures 

II 

Model  Requirement 
Functional  Spec. 

Document 

Pilot  Study 

6  Months 

3  People 

Requirements  Specification 
RAMSS  Design 

Model 

Risk  Assessment  Technique 
Output  Form 

RAMSS  Procedures 

Manual:  Data  Collection, 

OT&E  Evaluation,  RA, 
and  Reporting 

Pilot  Study 

Apply  RAMSS  to  Current 
or  Past  OT&E  Process 

Assess  Lessons  Learned 
Procedures 

Automated  Support 

III 

Upgrade  Model 
Concept  as 
Required  From 
Pilot  Study 
Develop  and 

Implement  Model 
Automated  Sup¬ 
port  Tools 

8  Months 

4  People 

Automated  Support  Tools 
for  RAMSS 

Selection  of  PDF 

Sensitivity  Analysis 
Bookkeeping 

Interface  to  S/W  Support 
Facility  and  S/W  Maint 
Tools 

Automated  Procedure  Support 

The 

reader  will  note  it 

is  proposed  that 

the  technique  selected  for 

development  be  experimented  with  manually  before  an  automated  model  is 
developed  and  implemented.  Due  to  the  nature  of  the  infancy  of  the 
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science  of  risk  assessment  of  software,  this  approach  should  help  to 
minimize  the  risk  of  developing  a  model  which  may  not  be  completely 
feasible. 

The  estimates  given  above  are  not  to  be  taken  as  final.  The  next 
report  of  this  subtask  will  examine  these  estimates  again  as  a  function 
of  the  further  analysis  of  candidate  risk  assessment  measures. 


,v 

\9 


III -35 


—  m  rn  r*  w  **  m  m  ^  *  *  to  »  •  ^  -  ■»  v  »  to  -m  -  »  ,  K  ."*•  k^n  _"V 


“  -  -  to  *  . 


THE  BDM  CORPORATION 


SECTION  IV 

RA  METHODOLOGIES,  TECHNIQUES,  TOOLS  TO  SUPPORT  AN  RAMSS 


4.1  TERMINOLOGY  AND  FOCUS. 


The  distinction  among  the  terms  methodology,  technique  and  tool  is 
not  always  clear.  In  fact,  entities  may  encompass  one  or  more  of  the 
terms.  Generally  a  methodology  is  a  broad-based  collection  of  methods, 
rules,  and  postulates  employed  by  a  particular  discipline.  A  technique 
is  one  of  the  methods  of  accomplishing  a  particular  aim.  A  tool  is  some¬ 
thing  (manual  or  automated)  which  facilitates  the  application  of  a  tech¬ 
nique.  Thus,  a  methodology  is  a  discipline-based  collection  of  tech¬ 
niques  and  tools.  Since  it  is  not  possible  to  always  make  such  a  clear 
distinction  as  to  which  category  a  given  entity  belongs,  this  section 
briefly  discusses  specific  entities  which  may  be  a  methodology,  tech¬ 
nique,  and/or  tool  for  risk  assessment.  The  term  "methodologies"  will 
be  used  henceforth  for  this  general  class  of  methodologies,  techniques, 
and  tools. 

Since  risk  is  the  potential  for  realization  of  unwanted,  negative 
consequences  of  an  event,  the  risk  assessment  focuses  upon  a  means  to 
present  that  "potential."  The  primary  means  is  as  a  probability.  Deter¬ 
mining  the  probability  across  possible  negative  consequences  of  an  event, 
and  across  applicable  events,  results  in  a  family  of  probability  func¬ 
tions  called  probability  density  functions  (PDFs).  These  PDFs  may  be 
discrete  or  continuous.  There  may  also  be  some  uncertainty  in  the  actual 
POF.  The  focus  of  risk  assessment  methodologies  is  upon  determining  a 
baseline  POF  representative  of  the  general  risk  function.  Then,  risk  is 
defined  when  a  measured  or  predicted  negative  outcome  value  is  compared 
to  the  baseline  density  function.  That  is,  risk  is  defined  by  those  out¬ 
comes  and  their  probabilities  that  are  negative  consequences  with  respect 
to  the  baseline.  With  this  approach,  ranges  of  risk  such  as  "risky," 
"not  risky,"  "high  risk,"  "low  risk,"  and  so  forth  can  be  reasonably 
quantified. 
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Risk  assessment  methodologies  generally  rely  on  subjectively-derived 
data  or  objectively-derived  data  from  the  applicable  area  (e.g.,  software 
supportabil ity) .  Subjectively-derived  data  depends  on  the  cognitive 
ability  of  human  beings  to  assign  weights  to  possible  -outcomes  of  an 
uncertain  event.  Objectively-derived  data  depends  on  empirical  observa¬ 
tions/measurements  and  associated  mathematical  relationships  among  the 
measured  (independent)  and  derived  (dependent)  variables  which  determines 
how  weights  should  be  assigned  to  outcomes  of  an  uncertain  event.  No 
matter  whether  subjective  or  objective  data  is  used,  an  underlying 
theoretical  foundation  for  probability  as  applied  to  risk  is  necessary  to 
interpret  and  determine  sensitivity  of  the  results. 

This  section  provides  a  brief  overview  of  the  theoretical  founda¬ 
tions  of  risk,  and  several  of  the  subjective-based  and  objective-based 
risk  methodologies.  The  information  presented  is  an  expansion  of  the 
information  presented  in  section  4.3  of  the  reference  5.1  report. 

4.2  THEORETICAL  FOUNDATION  FOR  RISK  MEASUREMENT. 

Risk  is  defined  as  "a  possible  negative  outcome"  (reference  5.30)  or 
as  "the  realization  of  unwanted,  negative  consequences  of  an  event" 
(reference  5.25).  These  definitions  imply  that  the  concept  of  risk  is 
two-dimensional;  i.e.,  risk  consists  of  two  parts.  One  part  of  risk  is 
the  negative  outcome  or  the  unwanted  consequence.  The  second  part  of 
risk  is  the  probability  or  potential  of  the  negative  outcome's  occur¬ 
rence.  These  two  parts  can  be  conveniently  thought  of  and  represented  as 
two  orthogonal  scales  as  shown  in  figure  4- 1(a). 

Probability,  the  vertical  scale  in  figure  4-l(a),  is  measured  in 
conventional  statistical  terms.  That  is,  the  measure  of  probability 
ranges  from  0  percent  (no  chance  of  occurrence)  to  100  percent  (absolute 
certainty  of  occurrence).  A  probability  value  is  associated  with  each 
outcome.  Outcome  can  be  measured  in  a  number  of  ways  and  depends  on  the 
problem  context  in  which  the  risk  assessment  is  being  made.  In  the  case 
of  software  supportabil ity,  outcome  may  be  specified  by  either  a  cost, 
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schedule,  or  performance  variable.  For  example,  consider  that  to  support 
a  given  software  package,  it  is  estimated  that  there  is  a  30  percent 
chance  that  supportabil ity  will  require  50,000  dollars,  a  50  percent 
chance  that  100,000  dollars  will  be  needed  for  supportabi 1 ity,  or  a 
20  percent  likelihood  that  150,000  dollars  will  be  required.  This  case 
is  depicted  in  figure  4-l(b). 

When  outcomes  are  assigned  probabilities  so  that  the  probabilities 
add  up  to  100  percent,  then  a  probability  density  function  is  estab¬ 
lished.  The  probability  density  function  is  a  fundamental  concept  to 
risk  assessment  and  its  estimation  is  the  basis  for  risk  determination. 
Probability  density  functions  may  be  discrete  (as  in  the  case  of  figure 
4-l(b))  or  continuous.  For  continuous  probability  density  functions,  the 
probability  of  occurrence  for  some  interval  of  outcomes  is  that  area 
under  the  density  function  that  is  cut  off  by  the  outcome  interval.  For 
example,  in  figure  4-l(c),  the  probability  of  an  outcome  greater  than  a 
and  less  than  b  is  the  area  under  the  curve  between  a  and  b. 

Implicit  to  the  definition  of  risk  is  the  notion  of  uncertainty.  If 
there  is  no  uncertainty,  there  is  no  risk.  Risk  analysts  do  not  discuss 
situations  with  certain  outcomes.  Risk  analysis  specifically  "attempt(s) 
to  quantify  uncertainty"  (reference  5.26),  and  it  is  the  probability 
density  function  that  is  the  vehicle  for  the  expression  of  uncertainty  in 
quantitative  terms.  From  the  example  depicted  as  figure  4- 1 ( b ) ,  it  is 
uncertain  as  to  the  cost  of  supporting  a  given  software  package.  The 
uncertainty  is  expressed  by  explicitly  stating  that  more  than  one  cost 
outcome  has  a  potential  for  occurrence.  In  other  words,  the  cost  of 
software  supportabil ity  is  not  certain.  Conversely,  if  it  is  certain 
that  software  supportabi 1 ity  will  require  100,000  dollars,  as  shown  by 
figure  4-l(d),  then  there  is  no  uncertainty  in  the  risk. 

Still  to  be  considered  in  the  definition  of  risk  is  the  negative 
aspect,  or  magnitude,  of  the  outcome.  The  concept  of  negative  outcome  or 
consequence  can  only  be  evaluated  with  respect  to  some  baseline  level. 
This  baseline  level  is  some  value  of  the  outcome  which  usually  represents 
the  available  resources.  For  instance,  if  we  are  allocated  120,000 
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dollars  for  supportabil ity  and  our  estimation  of  outcomes  are  those  shown 
in  figure  4-l(b),  then  the  negative  outcome  is  that  area  of  the  prob¬ 
ability  density  function  that  exceeds  the  baseline  value.  This  relation¬ 
ship  is  shown  in  figure  4-l(.e). 

Given  the  conceptualization  of  risk  put  forth  so  far,  then  qualita¬ 
tive  assessments  of  risk  can  be  directly  related  to  the  area  of  the 
probability  density  function  that  exceeds  the  baseline  value.  Now  terms 
such  as  "high"  or  "low"  risk  can  be  explicitly  defined  mathematically. 
As  an  example,  high  risk  may  be  defined  as  a  situation  where  40  percent 
or  more  of  the  probability  density  function  exceeds  the  baseline  value 
(figure  4-l(f)).  Low  risk  may  be  the  case  where  10  percent  or  less  of 
the  probability  density  function  exceeds  the  baseline  outcome  (figure 
4-1(9)). 

Risk  assessment  must  go  further  than  simply  considering  the  prob¬ 
ability  component  of  risk.  The  severity  of  the  outcome  has  to  be 
accounted  for.  To  illustrate  this  idea,  consider  figures  4-l(h)  and 
4-l(i).  In  both,  30  percent  of  the  probability  density  function  exceeds 
the  baseline  value.  However,  it  is  apparent  that  figure  4- 1 ( i )  repre¬ 
sents  the  riskier  situation  since  the  possible  outcomes  are  more  severe. 
Thus,  risk  is  some  combination  of  frequency  (probability)  and  severity. 

The  key  to  risk  assessment  is  the  estimation  of  the  probability 
density  function.  In  other  words,  some  estimate  must  be  made  of  the  out¬ 
comes  (e.g.,  costs)  and  the  probability  of  each  outcomes'  occurrence 
(perhaps  dependent  upon  risk  agent).  It  is  this  step  in  which  the  risk 
analyst  must  find  a  methodology  which  best  conforms  to  the  theoretical 
framework  of  risk  just  laid  out.  This  step  is  usually  an  arduous  task. 
Data  sources  for  most  risk  assessments  are  quite  limited.  Thus,  a  risk 
assessment  methodology  is  used  that  is  practical ,- implementable,  and  can 
yield  some  evaluation  of  risk,  however  partial  the  analysis.  Not  every 
risk  assessment  methodology,  however,  explicitly  or  implicitly  attempts 
to  estimate  a  probability  density  function. 
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4.3  RISK  METHODOLOGIES,  TECHNIQUES,  TOOLS. 

This  section  presents  several  risk  assessment  techniques  for  method¬ 
ologies)  that  have  appeared  in  the  literature.  Each  technique  is 
described  in  a  manner  to  give  the  reader  a  flavor  of  the  technique's 
mechanics,  advantages,  and  limitations.  A  comprehensive  description  of 
each  methodology  is  not  the  purpose  of  this  review.  Complete  details  of 
each  methodology  can  be  found  elsewhere:  Atzinger  and  Brooks  (reference 
5.26);  Megil  (reference  5.27);  Defense  Systems  Management  College 
(reference  5.35);  Apostolakis  (reference  5.36);  Behm  and  Vaupel 
(reference  5.37).  Often  entire  books  are  devoted  to  a  full  description 
of  some  of  the  methodologies  described  in  this  section. 

The  following  methodologies  are  described: 

a)  Choice-between  gambles  technique 

b)  Standard  lottery 

c)  Modified  Churchman-Ackoff  technique 

d)  Delphi  procedure 

e)  Closed  form  questionnaires 

f)  Bayesian  analysis 

g)  Network  analysis 

h)  Decision  trees 

i)  Parametric  model ing 

j)  Decision  theory. 

4.3.1  Subjective  Risk  Techniques. 

Risk  assessment  methodologies  usually  rely  on  either  objectively- 
derived  data  or  subjectively-derived  data.  First,  let's  consider  method¬ 
ologies  using  subjective  data.  Several  methodologies  exist  in  the 
literature  for  arriving  at  an  estimated  probability  density  function 
based  on  subjective  judgments.  These  methods  include:  choice-between- 
gambles  technique,  lotteries,  a  modified  Churchman-Ackoff  technique, 
modified  Delphi  technique,  8ayesian  estimates,  and  estimates  of  the 
moments  of  the  distribution  via  direct  questioning.  Several  other  risk 
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assessment  methodologies  exist  that  are  based  on  subjective  data.  How¬ 
ever,  none  of  these  other  methods  attempt  to  estimate  a  probability 
density  function.  These  methods  include  checklists,  qualitative  surveys, 
rating  scale  surveys,  and  so  on.  In  essence,  these  methods  attempt  to 
yield  a  "gut  feel"  of  risk  as  opposed  to  an  explicit  statement  of  risk  by 
a  probability  density  function.  Some  of  these  techniques  will  be 
described  in  some  detail  as  to  the  technique’s  mechanics,  advantages,  and 
1 i mi tations . 
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4. 3. 1.1  The  Difficulty  of  Making  Subjective  Estimates. 


Making  subjective  estimates  is  a  difficult  task  for  humans  to 
handle.  In  other  words,  subjective  estimates  place  severe  demands  on  the 
cognitive  abilities  of  human  beings.  Because  humans  have  limited  abili¬ 
ties  to  process  information  and  solve  problems,  estimation  by  subjective 
means  must  be  evaluated  in  this  light. 

Because  of  the  complexity  and  difficulty  of  many  information¬ 
processing  and  decision-making  situations,  the  human  mind  employs 
"heuristics."  These  heuristics  or  "rules  of  thumb"  simplify  the  com¬ 
plexity  of  a  given  task.  One  heuristic  that  has  been  identified  is 
anchoring  and  adjustment.  An  anchor  is  some  original  point  estimate. 
For  example,  say  that  one  is  estimating  the  cost  of  a  software  package 
that  is  to  be  developed.  One's  expected  cost  may  be  100,000  dollars. 
This  initial  anchor  is  now  used  to  assess  various  other  estimates  of  the 
cost,  say  the  lowest  possible  cost  or  the  highest  possible  cost.  How¬ 
ever,  the  original  estimate  of  the  expected  value  (100,000  dollars)  is 
such  a  heavy  anchor  that  adjustments  around  the  point  estimate  are  too 
small.  That  is,  the  lowest  possible  cost  estimate  and  the  highest  pos¬ 
sible  cost  estimate  are  too  close  to  the  "anchor"  estimate.  In  other 
words,  people's  probability  density  functions  are  too  tight.  This  holds 
even  if  the  initial  estimate  is  no  more  than  a  guess. 

The  estimation  of  a  median  value  of  some  uncertain  quantity  displays 
other  limitations  to  the  human  mind.  People  tend  to  focus  their  thinking 
on  the  unique  situational  features  of  the  particular  case  facing  them. 
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The  information  obtained  in  the  actual  outcomes  of  similar  cases  is  often 
ignored.  There  is  a  tendency  to  mentally  enhance  some  information  of 
certain  factors  in  order  to  increase  the  perceived  uniqueness  of  the 
problem  at  hand.  Each  of  us  likes  to  believe  that  his/her  own  situation 
is  unique.  However,  to  pretend  that  one's  own  situation  is  so  unique 
that  it  has  no  relationship  to  similar  experiences  is  foolish. 

Another  heuristic  employed  by  the  human  mind  is  availability.  When 
an  individual  assesses  the  probability  that  an  event  will  occur  by 
imaging  similar  events  or  recalling  related  information,  they  are  using 
the  availability  heuristic.  The  more  "available"  such  related  instances 
or  information  are,  the  higher  the  probability  assigned  to  that  event. 
As  an  example,  if  one  saw  a  fatal  car  accident  yesterday,  then  today 
their  estimate  of  the  probability  of  a  fatal  accident  would  be  artifi¬ 
cially  high.  The  availability  heuristic  helps  explain  why  the  time  or 
cost  needed  to  complete  a  task  is  usually  underestimated.  There  is,  of 
course,  an  incentive  for  making  a  low  estimate  when  doing  the  work  for 
someone  else,  but  even  when  people  are  making  the  estimate  for  them¬ 
selves,  the  bias  towards  estimates  that  are  too  low  still  exists.  The 
most  "available"  scenario  is  the  "surprise-free"  one--each  subtask  is 
easily  completed  in  the  minimum  time--and  thus  the  median  estimate  is 
usually  derived  from  the  no-complication  scenario. 

4. 3. 1.2  Choice-8etween-Gambles  Technique. 
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The  objective  of  the  choice-between-gambles  technique  is  to  subjec¬ 
tively  derive  a  discrete  probability  density  function.  The  density  func¬ 
tion  displays  the  probability  that  a  component  characteristic  (e.g.,  a 
cost,  schedule,  or  performance  variable)  will  achieve  a  specified  level. 
The  inputs  for  eliciting  these  probability  responses  from  a  group  of 
experts  are  composed  of  choice-between-gambles  or  betting-type  questions 
administered  by  the  analyst.  It  is  believed  by  many  authors  in  the  field 
of  subjective  decision  making,  that  this  form  of  questioning  results  in  a 
more  realistic  probability  density  function  than  a  direct  questioning 
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approach.  This  latter  method  of  asking  the  expert  directly  what  prco- 
ability  he/she  attaches  to  a  particular  outcome,  although  simple  in 
application,  has  little  likelihood  of  success  according  to  many  authors. 
Many  individuals  either  have  no  ability  to  think  directly  in  terms  of 
probability,  or  they  have  difficulty  communicating  the  probabilities 
without  the  aid  of  some  tool  such  as  the  choice-between-gambles 
technique. 

4. 3. 1.2.1  Description. 


The  choice-between-gambles  technique  is  an  iterative  procedure  which 
is  initiated  by  presenting  two  alternative  gambling  situations  to  an 
expert.  The  expert  is  asked  to  choose  between  (11  a  real-world  gamble 
involving  values  of  a  component  characteristic  (e.g.,  cost)  of  the 
project  in  question  with  unspecified  probabilities  and  (2)  a  hypothetical 
gamble  involving  two  objective  events,  El  and  E2,  with  given  probabili¬ 
ties,  P(E  1)  and  P(E2).  The  monetary  payoffs  for  both  gambles  are  made 
equal  to  facilitate  the  expert's  ability  to  discriminate.  Next,  the 
probabilities  of  the  hypothetical  gamble  are  varied  (starting  with  equal 
probabilities  for  El  and  E2)  until  the  expert  is  indifferent  between  the 
two  gambles.  Hence,  the  expert's  subjective  probabilities  regarding  the 
outcomes  of  the  real-world  gamble  are  inferred  by  the  resulting  prob¬ 
abilities  from  the  hypothetical  gamble. 

As  an  illustration  of  this  technique,  consider  a  real-world  gambling 
situation  involving  two  possible  costs  for  an  avionics  software  package 
and  a  hypothetical  gamble  with  possible  events  El  and  E2.  The  payoffs 
are  stated  as:  (1)  $10  if  one  cost  is  realized,  and  SO  if  the  other  cost 
occurs;  and  (2)  $10  if  event  El  occurs,  arid  $0  if  event  E2  is  realized. 
Table  4-l(a)  reflects  this  initial  decision  situation. 

The  assumption  is  that  if  the  expert  chooses  the  real-world  gamble, 
he  will  receive  $10  if  the  sof‘.:are  cost  of  36,000  dollars  (plus  or  minus 
1,000  dollars)  actually  occurs.  The  expert  will  receive  $0  if  any  other 
cost  is  realized.  If  the  expert  selects  the  hypothetical  gamble,  he  will 
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Table  4-1. 

Example: 

Choice  Between  Gambles  Technique 

(a)  Decision  Situation 

Payoff 

Cost 

Probabil ities 


Real-World  Gamble 


S 

$36, 0< 
$+1,01 
7 


Hypothetical  Gamble 


0 

Payoff 

$10 

0 

not  $36,000 
+1,000 

Event 

E1 

E 

7 

Probabi 1 ities 

0.5 

0 

(b)  Revised  Decision  Situation 


Real-World  Gamble 


Hypothetical  Gamble 


Consequence 

Cost 

Probabilities 


$ 

$36,0i 

$+1,0 


I 


0 

Consequence  $10 

not  $36,000 

Event  E^ 

$+1,000 

7 

Probabi 1 i ties  0.7 

(c)  Final  Probability  Distribution 


$32,000  +1,000 
$34,000  +  1,000 
$36,000  +  1,000 
$38,000  +  1,000 
$40,000  +  1,000 


°robabi 1 ity 
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receive  S10  if  El  occurs  and  $0  if  E2  occurs.  Therefore,  if  the  expert's 
decision  in  the  first  round  is  to  accept  the  real-world  gamble,  then  it 
is  immediately  inferred  that  his/her  subjective  probability  assessment 
that  a  cost  of  36,000  dollars  (plus  or  minus  1,000  dollars)  will  be 
achieved  is  greater  than  0.5.  Thus,  in  the  next  decision  rounds  the 
analyst  will  adjust  the  probability  of  occurrence  of  the  hypothetical 
event  El  upward,  and  that  for  event  E2  downward.  This  procedure  is  then 
continued  in  an  iterative  fashion  to  the  stage  where  the  expert  is 
indifferent  to  the  two  gambling  situations.  Suppose  that  this  stage  is 
ultimately  achieved  at  P ( El )  3  0.7,  P(E2)  3  0.3.  Then  P(software  cost  = 
36,000  dollars  plus  or  minus  1,000  dollars)  is  inferred  to  be  0.7.  This 
revised  decision  situation  is  depicted  in  table  4-l(b). 

Having  obtained  the  probability  of  software  cost  equal  to  36,000 
dollars,  the  next  step  is  to  change  the  software  cost  in  the  real-world 
gamble  to  the  next  interval  that  the  expert  will  be  able  to  discriminate 
between  its  probability  of  occurring  over  that  of  the  previous  value.  At 
each  successive  stage,  then,  probabil ities  are  derived  for  various 
interval  values  of  software  cost.  Finally,  each  endpoint  of  the  prob¬ 
ability  density  function  is  determined  when  the  expert  is  indifferent 
between  the  two  gambles,  with  P ( El )  =0  and  P(E2)  3  1. 

The  resulting  probability  density  function  for  this  example  could  be 
shown  as  in  table  4-l(c). 

Notice  in  table  4- 1 (c )  that  the  total  of  the  subjective  probabili¬ 
ties  do  not  equal  1.0;  instead  the  total  is  1.1.  Given  this  situation, 
then  the  analyst  can  either:  (1)  reassess  the  expert’s  subjective  prob¬ 
abilities,  or  (2)  normal  i  ze  the  derived  probabilities  so  that  the  total 
equals  1.0. 


4. 3. 1.2. 2  Advantages. 

The  choice-between-gambles  technique  derives  probability  density 
functions  through  inference  rather  than  by  direct  questioning.  As  noted 
earlier  in  the  discussion,  it  appears  that  such  an  organized  stepwise 
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procedure  for  eliciting  a  probability  density  function  results  in  a  more 
val id  output. 

Compared  to  most  other  techniques,  the  choice-between-gambles  tech¬ 
nique  is  not  time  consuming  in  its  application.  It  is  simple  to  apply 
and  results  directly  in  a  probability  density  function. 

4. 3. 1.2. 3  Limitations  . 

First,  the  choice-between-gambles  technique  only  results  in  discrete 
probability  density  functions.  Continuous  probability  density  functions 
cannot  be  obtained  as  the  technique  is  described  here.  Second,  the 
expert  may  find  it  difficult  to  determine  the  highest  or  lowest  value  for 
which  he/she  can  state  a  subjective  probability. 


4. 3. 1.2.4  Assumptions. 


It  is  assumed  that  the  monetary  rewards  offered  as  consequences  for 
correct  responses  are  sufficient  in  magnitude  to  motivate  the  expert  in 
forming  his/her  judgements.  It  is  also  assumed  that  the  expert's  concern 
for  the  project  success,  his/her  integrity,  and  his/her  decision-making 
abilities  contribute  to  the  degree  to  which  his/her  judgements  represent 
his/her  personal  beliefs.  Finally,  it  is  assumed  that  the  so-called 
expert  is  in  fact  knowledgeable  and  experienced  in  his/her  field.  The 
expert  must  also  be  sufficiently  well-founded  in  probability  theory  to 
respond  meaningfully  to  the  questioning  procedure. 


4.3. 1.3  The  Standard  lotter' 


The  objective  of  the  standard  lottery  technique  is  the  derivation  of 
a  probability  density  function  over  all  possible  values  of  a  given  com¬ 
ponent  characteristic  (e.g.,  cost).  The  procedure  involves  presenting 
the  expert  with  two  gambling  situations.  This  technique  differs  from  the 
choice-between-gambles  method  in  that  it  does  not  involve  the  process  of 
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varying  actual  probabilities  until  indifference  is  achieved.  Instead, 
numbers  representing  randomly  selected  lottery  tickets  from  a  batch  of 
100  are  varied  in  an  attempt  to  achieve  indifference.  In  essence,  the 
number  of  such  tickets  directly  infers  component  risk  probabilities. 


4. 3. 1.3.1  Description. 

The  standard  lottery  technique  is  based  on  the  following  basic 
lottery  description.  In  a  lottery,  a  contestant  purchases  as  many 
tickets  as  desired.  The  more  tickets  he/she  purchases,  the  greater  his/ 
her  chance  of  winning  the  contest  prize.  After  the  purchase  of  tickets 
is  completed,  one  number  is  randomly  drawn  from  a  lot  of  (for  example) 
100  equally  likely  numbers.  That  is,  each  contestant  fully  understands 
that  any  number  between  1  and  100  has  an  equal  chance  of  being  selected. 
The  winning  contestant  is  that  individual  who  owns  a  lottery  ticket  with 
the  number  on  it  coinciding  with  the  number  selected.  For  example,  a 
contestant  might  have  purchased  40  tickets;  thus  his/her  chance  of 
winning  the  contest  prize  is  0.4. 

The  standard  lottery  technique  proceeds  as  follows: 

a)  Specify  a  possible  component  characteristic  value  (e.g., 
the  cost  of  an  avionics  software  package  is  100,000 
dollars)  for  the  real-world  event. 

b)  Direct  the  expert  to  imagining  that  he/she  is  given  a 
choice  between  a  certain  number  of  tickets  in  the  standard 
lottery  with  a  prize  of  value  V  (e.g.,  $10)  and  the  right 
to  receive  the  same  prize  if  the  real-world  event  is 
real i zed. 

c)  For  a  given  initial  number  of  lottery  tickets  (e.g.,  30) 
ask  the  expert  which  alternative  gamble  he/she  feels  has 
the  greatest  chance  of  winning  the  prize:  (1)  the  reali¬ 
zation  of  the  real-world  event,  or  (2)  the  holding  of  the 
specified  number  of  tickets  (i.e.,  30)  of  a  lottery  of  100 
tickets  total . 
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d)  If  one  gamble  is  preferred  over  the  other,  then  vary  the 

given  number  of  tickets  and  repeat  step  c. 

e)  Repeat  steps  c  and  d  until,  in  the  expert's  opinion, 

he/she  feels  that  the  possibility  of  receiving  the  prize 
in  the  event  has  exactly  the  same  likelihood  as  some 

number  of  tickets  (say  70)  in  the  lottery.  Thus,  it  is 

inferred  that  the  expert  considers  that  there  is  a 

70  percent  chance  of  the  software  cost  being  100,000 

dollars. 

f)  Employing  steps  a  through  e,  the  expert  can  proceed 

analogously  to  assign  probabilities  to  all  other  possible 
real-world  events . 

Similar  to  the  choice-between-gambles  technique,  the  probabilities 
must  sum  to  1.0.  Normalization  can  be  used  in  those  cases  where  the 
probabilities  do  not  exactly  sum  to  unity. 


4. 3. 1.3.2  Assumptions,  Limitations,  Advantages. 

The  standard  lottery  technique  also  provides  an  improved  process  for 
eliciting  subjective  responses  over  direct  questioning.  The  lottery 
technique  is  similar  to  the  choice-between-gambles  technique  and  thus 
offers  the  same  advantages,  limitations,  and  assumptions.  Importantly, 
the  expert  with  little  probability  theory  background  may  be  more  com¬ 
fortable  with  this  technique  as  opposed  to  the  choice-between-gambles 
method. 


» 


4.3. 1.4  The  Modified  Churchman-Ackoff  Technique. 


The  modified  Churchman-Ackoff  technique  differs  from  the  previous 
two  methods  in  that  (1)  there  are  no  betting  situations,  and  (2)  the 
expert  is  instead  asked  to  make  "greater  than,"  "equal  to,"  or  "less 
than"  evaluations  regarding  relative  probabilities.  A  resulting  relative 
probability  scale  is  easily  transformed  into  a  probability  density 
function. 
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4. 3. 1.4.1  Description. 

In  the  modified  Churchman-Ackoff  technique,  the  expert  must  reveal  a 
range  of  possible  values  which  the  component  characteristic  (e.g.,  the 
cost  of  a  software  package)  could  possibly  realize.  By  employing  some 
other  technique,  end  point  values  of  zero  probability  of  occurrence  must 
first  be  specified.  These  values  need  only  be  any  low  and  high  values 
which  the  expert  specifies  as  having  zero  probability  of  occurrence  in 
the  proposed  system. 

Next,  individual  values  within  the  range  of  possible  values  must  be 
determined.  These  values,  which  will  form  the  set  of  comparative  values 
(e.g.,  cost  values)  for  this  technique,  are  specified  by  the  following 
approach: 

a)  Start  with  the  smallest  value. 

b)  Progress  upward  on  the  scale  of  values  until  the  expert  is 
able  to  state  a  simple  preference  regarding  the  relative 
probabi 1 ities  of  occurrence  of  the  two  characteristic 
values.  If  he/she  is  able  to  say  that  he/she  believes  one 
value  has  either  a  greater  chance  or  a  lesser  chance  of 
occurring  than  the  other  of  the  two  values,  then  it  is 
inferred  that  the  expert  is  able  to  discriminate  between 
the  two  values. 

c)  Using  the  higher  of  the  two  previously  specified  scale 
values  as  a  new  basis,  repeat  step  b  to  determine  the  next 
value  on  the  scale. 

d)  Repeat  steps  b  and  c  until  the  high  end  point  value  of  the 
range  of  parameter  values  is  approached. 

Employing  this  procedure,  one  might  obtain  the  results  in  table 
4-2(a). 

The  descending  order  of  probability  of  occurrence  can  be  determined 
by  applying  the  following  paired  comparison  method. 

Ask  the  expert  to  compare,  one  at  a  time,  the  first  discrete  value 
(e^)  of  the  set  to  each  of  the  other  values  (9,,,  9^,  etc.).  The  expert 
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Table  4-2. 

Example:  Modified  Churchman-Ackoff  Technique 

(a)  Characteristic  Values  for  Software  Cost 

91  =  $35,000 
02  =  $36,000 
03  =  $37,500 
04  =  $38,500 
05  =  $40,000 
9g  =  $41,000 
07  =  $41,500 

(b)  Paired  Comparisons 


0^  vs  9^ 0^ 


93  <94 
93>95 

V9« 

S3  >97 


0  vs  0  0 

4  V5  5  7 


3.  >  ®5 
4  5 

94>96 

®4  >97 


95  vs  V  ®7 


®5>96 

05>97 


(c)  Summary  of  Preference  Relationships 


=  6  times 


=  5  times 
=  4  times 
=  3  times 
=  2  times 
=  0  times 
=  0  times 
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96  vs  ®7 


0  >  0 
6  7 
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Table  4-2. 

Example:  Modified  Churchman-Ackoff  Technique  (Continued) 


Characteristic  Value 

$38,500  94 
$37,500  03 
$40,000  95 
$36,000  92 
$41,000  9g 
$35,000  9j_ 
$41,500  By 


(d)  Transformation 


Preference  Rank 


New  Symbol 


(e)  Relative  Probability  Ratings 

RX^  =  100  probability  points 

RXy  =  30  probability  points 

RX3  =  50  probability  points 

RX^  =  25  probability  points 

RXg  =  10  probability  points 

RXC  =  0  Drobability  points 

0 

RX,  =  0  probability  points 
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Table  4-2. 

Modified  Churchman-Ackoff  Technique  (Concluded) 


(f)  Probability  Density 


Component 

Characteristic 

Value 
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is  asked  to  state  a  preference  for  that  value  in  each  group  of  two  values 
that  he/she  believes  has  the  greater  chance  of  occurring.  In  other 
words,  the  expert  chooses  one  value  which  has  the  greatest  chance  of 
occurrence  for  each  paired  comparison.  The  following  hypothetical 
preference  relationships  could  result  for  the  set  of  7  values: 

{9^  <  02»  9^  <  9^ ,  9^  <  9g»  9^  <  9g ,  9^  <  9 -j  }■ 

Next,  ask  the  expert  to  compare,  one  at  a  time,  the  second  discrete 
value  (62)  of  the  set  to  each  of  the  other  values  succeeding  it  in  the 
set  (i.e.,  0^,  04,  etc.).  The  following  preference  relationships  might 
result: 

{@2  ^  ®3’  ^  84*  ®2  ^  9s»  0£  ^  9g*  ^  9y} ' 

Continue  the  process  until  all  values  (9^)  have  been  compared  to  the 
others.  For  example,  table  4-2(b)  lists  preferences  which  might  result 
for  the  remaining  cost  values. 

Now  total  the  number  of  times  (0^)  value  was  preferred  over  other 
values.  The  results  for  this  procedure  are  listed  in  table  4-2(c). 

List  the  values  in  descending  order  of  simple  ordinal  probability 
preference  and  change  the  symbols  for  each  value  from  9  to  X(j)  as  shown 
in  table  4-2(d) . 

Arbitrarily  assign  a  rating  of  100  points  to  the  characteristic 
value  (e.g.,  cost)  with  the  highest  subjective  probability.  Then,  as  in 
the  first  step,  question  the  expert  regarding  the  relative  chance  of 
occurrence  of  each  of  the  other  values  on  the  ordinal  scale  in  table 
4-2(d)  with  respect  to  the  value  at  the  top  of  the  scale.  Assigning  X(l) 
a  rating  of  100  points,  the  expert  is  first  questioned  as  to  his/her 
feeling  of  the  relative  chance  of  occurrence  of  the  second  highest  scale 
value  (e.g.,  X(2))  with  respect  to  X(l)).  Does  it  have  a  25  percent  as 
much  chance  of  realization  as  X(l)?  A  60  percent?  A  70  percent?  The 
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relative  probability  rating,  based  on  100  points,  (i.e.,  100  percent  as 
much  chance)  will  then  be  posted  for  X(2).  For  example,  if  the  expert 
decides  that  X(2)  has  8/10  as  much  chance  of  occurring  as  does  X ( 1 ) ,  the 
ratings  become  X(l)  =  100  points  and  X ( 2)  =  30  points. 

Next,  question  the  expert  about  the  relative  chance  of  occurrence  of 
the  next  highest  scale  (e.g.,  X(3)),  first  with  respect  to  the  most  pre¬ 
ferred  value  (X(l)),  and  second  with  respect  to  the  second  most  preferred 
scale  value  (X(2))_  The  resulting  numerical  ratings  should  concur. 

If  the  expert  expresses  a  belief  that  X(3)  has  1/2  as  much  chance  as 
X ( 1 )  and  5/8  as  much  chance  as  X(2)  (as  a  validity  check),  this  confirms 
that  the  relative  probability  of  occurrence  rating  for  X(3)  is  50.  The 
scale  now  is  X(l)  =  100  points,  X(2)  =  80  points,  and  X(3)  =  50  points. 

The  process  is  continued  for  each  remaining  successively  lower  scale 
value  on  the  ordinal  scale  shown  in  table  4-2(d).  Determine  the  relative 
number  of  points  to  be  accorded  each  value  with  respect  to  the  top  scale 
value  and  with  respect  to  all  other  values  on  down  the  scale  which  are 
above  the  characteristic  value  in  question. 

In  the  event  of  minor*  disparities  between  relative  probability 
ratings  for  a  given  value,  the  average  of  all  such  ratings  for  that 
characteristic  value  (e.g.,  cost  value)  might  be  computed.  For  example, 
X(4)  might  be  determined  to  be  3/10  as  probable  as  X(l),  1/4  as  probable 
as  X(2) ,  and  1/2  as  probable  as  X(3).  The  three  absolute  ratings  for 
X(4)  are  thus  inferred  to  be  30,  20,  and  25  points  respectively.  The 
average  of  these  ratings  is  25.  However,  before  averaging  such  figures, 
it  might  be  beneficial  to  have  the  expert  reevaluate  his  relative  ratings 
for  X(4)  with  respect  to  X(l),  X(2),  and  X(5). 

As  a  result  of  the  above  process,  the  relative  probability  values 
shown  in  table  4-2(e)  might  be  attained. 

Finally,  the  scale  of  relative  probability  values  can  be  converted 
directly  into  a  scale  of  actual  probability  density  values  by  letting 
P(Xj)  equal  the  actual  subjective  probability  of  occurrence  of  the 
highest  value.  Then,  P(X2)  is  then  defined  as 
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R(x?) 

p(xi}- 

Similarly  P^ )  is  defined  as 

R(X.) 

RlJii  P(X1)* 

for  i  =  2,  3,. ..,7. 

Assuming  that  the  independent  characteristic  values  evaluated 
represent  all  possible  values  attainable  by  the  component  characteristic, 
the  respective  probabilities  must  sum  to  1.0  (i.e.,  P(Xh  +  P(X2)  + 


P(X3)  +  P(X4)  +  P(X-)  *  P(X6)  +  P(X7 

)  *  1.0). 

Substituting  the  expres- 

sions  for  P ( X^ ) ,  i  =  2,... 

,7,  it  follows  that 

R(X2) 

R(X,) 

r(x4) 

p(V  +  ^  p(V 

*  irnqT 

P(XX)  + 

P(Xj) 

R(Xg) 

R(*S> 

RfX7) 

P<V 

+  WZJ 

P(XX)  + 

rTx^T 

P(XT)  =  1.  ■ 

Solving  this  equation  for 

P(x{).  the 

remaining 

P(Xi), 

i  =  2,. ..,7  can  be 

determined  using  the  relationship 

R(X,) 

p<V*r*iT  P(V- 

As  an  illustration,  consider  the  relative  probability  ratings  in 
table  4-2(e).  Using  these  values,  the  preceding  equation  is  given  by 

P(xi)  +  p(xi}  +  P(V  +  m  P(V  +  T3o  ?(V  3  l- 


Solving  this  equation,  P(X^)  =  0.377. 

This  value  can  be  used  to  determine  the  remaining  probabilities  as 
follows: 
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=  0.80(0.377)  =  0.301 

=  0.50( .0377)  =  0.189 

*  0.25(0.377)  =  0.0095 

=  0.10(0.377)  =  0.038 

=  0(0.377)  =  0.000 

=  0(0.377)  =  0.000 

The  resulting  probability  density  appears  in  table  4-2(f). 

4. 3. 1.4. 2  Advantages . 

The  modified  Churchman-Ackoff  technique  offers  an  alternative  to  the 
two  previous  methods  of  eliciting  absolute  subjective  probability 
responses.  In  this  case,  relative  probabilities  with  respect  to  one 
chosen  most  probable  characteristic  value  are  derived.  In  some  situa¬ 
tions,  the  expert  may  think  it  easier  to  make  evaluations  with  respect  to 
a  characteristic  state  that  he/she  feels  has  the  greatest  possibility  of 
realization. 

In  addition,  this  technique  offers  a  systematic  method  of  checking 
the  consistency  of  relative  value  judgements  made  by  the  experts.  This 
enhances  the  validity  of  the  resulting  probability  distribution  function. 

4.3. 1.4.3  Limitations . 

The  modified  Churchman-Ackoff  technique  does  not  involve  betting 
situations  which  are  generally  considered  more  successful  in  eliciting 
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correct  responses.  Instead,  it  involves  an  untested  approach  of  directly 
eliciting  relative  percentage  chances  of  occurrence  statements  for  each 
value  with  respect  to  the  occurrence  of  other  characteristic  values 
(e.g.,  does  a  software  cost  of  100,000  dollars  have  half  as  much,  or 
70  percent,  or  90  percent  as  much  chance  of  occurring  as  120,000 
dollars) . 

As  with  the  other  techniques  discussed  so  far,  the  probability 
values  are  still  judgements.  That  is,  of  course,  the  limitation  of  all 
techniques  involving  subjective  (as  opposed  to  objective)  decision 
making. 


4. 3. 1.5  The  Delphi  Procedure. 

Historically,  the  approach  for  obtaining  a  group  consensus  has  been 
the  formation  of  committees,  commissions,  or  councils.  While  the  basic 
philosophy  may  be  sound,  committees  tend  to  pressure  individuals  into 
conforming.  In  addition,  all  opinions  may  not  be  expressed  because  of 
the  personalities  of  the  individuals  and/or  because  of  the  relationship 
of  the  individuals  within  the  group.  Another  drawback  of  committees  is 
the  tendency  to  spend  a  great  deal  of  time  discussing  irrelevant  issues. 
Further,  irrelevant  information  may  degrade  the  group's  opinion.  More 
serious  though,  is  the  possibility  of  a  complete  breakdown  of  the  com¬ 
mittee.  That  is,  there  is  an  inability  of  the  committee  to  arrive  at  a 
general  consensus. 

The  Delphi  procedure  is  an  alternative  to  the  conmittee  approach  for 
eliciting  a  group  judgement.  The  Delphi  method  attempts  to  improve  the 
panel  or  committee  approach  in  arriving  at  a  forecast  or  estimate  by  sub¬ 
jecting  the  views  of  individual  experts  to  each  other's  criticism  in  ways 
that  avoid  face-to-face  confrontation.  Thus,  there  is  anonymity  of 
opinions  and  of  arguments  advanced  in  defense  of  these  opinions.  Direct 
debate  is  replaced  by  the  interchange  of  information  and  opinion  through 
a  carefully  designed  sequence  of  questionnaires.  The  participants  are 
asked  not  only  to  give  their  opinions,  but  the  reason  for  the  opinions. 
At  each  successive  questioning  session,  the  experts  are  given  new  and 
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refined  Information,  In  the  form  of  opinion  feedback,  which  is  derived  by 
a  computed  consensus  from  earlier  parts  of  the  program.  The  process  con¬ 
tinues  until  further  progress  toward  a  consensus  appears  to  be 
negligible.  The  conflicting  views  are  then  documented  along  with  the 
consensus . 

In  sun,  the  primary  features  of  Delphi  procedures  are: 

a)  Anonynity  of  the  sources  of  information  among  experts. 

b)  Iteration  with  controlled  feedback  of  group  responses  from 
iteration  to  iteration. 

c)  Statistical  group  response  (e.g.,  a  median  value). 

4. 3. 1.5.1  Description. 

The  steps  of  the  procedure  for  estimating  a  group  probability 
density  function  are  outlined  below  for  the  cost  of  an  avionics  software 
package. 

Employing  the  first  two  steps  of  the  modified  Churchman-Ackoff  tech¬ 
nique,  each  expert  is  asked  to  reveal  his  estimate  of  the  total  range  of 
values  which  the  software  cost  could  realize.  The  individual  values 
within  this  range  will  form  the  sets  of  comparative  cost  values.  Then 
list  all  cost  values  specified  by  all  of  the  experts.  These  will  form 
the  list  of  cost  values  to  be  investigated,  for  example,  as  those  shown 
in  table  4-3(a).  . 

The  list  of  characteristic  values  (e.g.,  cost  values)  to  be  investi¬ 
gated  are  included  in  table  4-3(b). 

In  the  first  round,  randomly  select  a  cost  value  from  the  list  in 

table  4-3(b)  and  ask  each  expert  to  give  an  independent  estimate  of  its 

probability.  Each  expert  is  questioned  alone.  In  addition,  each  expert 
is  asked  his/her  reasons  regarding  the  probability  assessment. 

Arrange  the  probability  responses  from  all  experts  in  order  of  mag¬ 
nitude,  and  determine  its  quartiles,  Ql,  M,  Q3,  so  that  approximately  one 

quarter  of  all  estimates  lie  in  each  interval.  For  example,  for  the 

selected  software  cost  of  31,000  dollars,  the  probabilities  for  experts 
El,  E2,  E3,  E4,  and  E5  might  occur  as  shown  in  table  4-3(cl. 
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Table  4-3. 

Example:  Delphi  Procedure 
(a)  Possible  Characteristic  Values  by  Experts 


Expert  1 

($) 

Expert  2 

($) 

Expert  3 

($) 

Expert  4 

($) 

Expert 

($) 

28,000 

29,000 

30,000 

23,000 

30,000 

31,000 

31,000 

32,000 

30,000 

32,000 

32,000 

33,000 

34,000 

32,000 

34,000 

34,000 

36,000 

37 ,000 

33,000 

36,000 

37 ,000 

40,000 

39,000 

36,000 

37,000 

(b)  List  of  Possible  Discrete  Characteristic  Values 


$28,000 

$29,000 

S30.000 

$31,000 

$32,000 

$33,000 

$34,000 

$36,000 

$37,000 

$39,000 

$40,000 


THE  BOM  CORPORATION 

Table  4-3. 

Example:  Delphi  Procedure  (Concluded) 


EXPERT  E4  E2  E1  Eg  E3 

PROBABILITY  .15  .25  .3  .3251  .4 


EXPERT 

a 

E2 

E1 

E5 

£3 

PROBABILITY 

.25 

|2j| 

.3 

.325 

.35 

(d)  PROBABILITY  RESPONSES:  2ND  ROUND 


EXPERT 

E2 

E4 

E1 

E5 

PROBABILITY 

.275 

.275 

.3 

.325 
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Reveal  the  values  and  responses  of  each  interval  to  each  member,  and 
if  his/her  estimate  lies  outside  the  first  round  interquartile  range,  Q1 
to  Q3,  then  ask  the  expert  to  state  reasons  why  the  answer  should  be 
lower  (or  higher)  than  that  of  the  75  percent  majority  opinion  expressed 
in  the  first  round. 

Give  these  new  responses  back  to  all  respondents  by  communicating 
the  new  range  of  the  new  quartile  values,  along  with  independently  stated 
reasons  for  the  estimates  outside  the  75  percent  majority  opinion.  The 
experts  are  now  asked  to  consider  the  reasons  given,  weigh  their  feasi¬ 
bility,  and  revise  their  own  previous  estimates  accordingly.  For  the 
software  cost  example,  assume  that  the  second  round  scale  is  as  shown  in 
table  4-3(d). 

If  the  newly  revised  probabilities  still  fall  outside  the  second 
round  interquartile  range,  respondents  are  asked  to  state  why  they  found 
previous  arguments  unconvincing  enough  to  draw  them  toward  the  median. 

In  the  third  round,  the  quartile  results  of  round  2  are  submitted  to 
respondents  along  with  the  counter  arguments  elicited.  These  respondents 
are  then  asked  to  make  a  final  revision  of  their  estimates. 

The  mean  value  of  the  resulting  round  3  estimates  (table  4-3(e))  is 
taken  as  the  group  response  as  to  what  the  subjective  probability  con¬ 
sensus  for  the  software  cost  value  should  be.  For  this  example,  the  mean 
third  round  subjective  estimate  for  a  cost  of  31,000  dollars  is 

2(0.275)  +  (0.3000)  +  2(0.325)  /5  =  0.300  =  P(Cost  =  $31,000) 

Now  repeat  the  procedure  for  a  second  possible  cost  value.  Normalize  the 
distribution  if  necessary. 

4. 3. 1.5. 2  Advantages. 

In  situations  where  one  wants  to  use  group  judgement  to  analyze 
uncertainty,  the  Delphi  procedure  provides  an  alternative  to  the  com¬ 
mittee  approach  in  the  identification  and  consolidation  activities  of  a 
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risk  analysis.  The  Delphi  method  attempts  to  improve  upon  the  committee 
approach  by  allowing  the  exchange  of  information  in  an  environment  that 
reduces  the  group  pressure  to  conform.  Also  there  is  the  removal  of  the 
impact  of  the  dominant  individual. 

4.3. 1.5.3  Limitations . 

Perhaps  the  biggest  drawback  in  applying  the  Delphi  procedure  at 
this  time  is  that  very  few  analysts  have  experience  in  using  the  tech¬ 
nique.  In  particular,  there  is  a  lack  of  training  in  preparing  question¬ 
naires  and  analyzing  the  results.  One  should  not  discount  the  importance 
of  such  training.  If  the  questionnaire  is  prepared  by  unqualified 
people,  the  answers  to  the  questions  may  be  biased  or  the  questions  them¬ 
selves  may  not  really  address  the  problem.  In  addition,  since  the  pro¬ 
cedure  has  had  limited  exposure,  it  may  not  be  accepted  immediately. 

Another  important  consideration  in  the  selection  of  the  Delphi  pro¬ 
cedure  is  time;  both  time  available  for  conducting  the  analysis  and  time 
involved  in  applying  the  procedure.  Clearly,  if  there  is  little  time 
available  for  developing  a  consensus  of  opinion,  then  the  Delphi  tech¬ 
nique  may  not  be  a  viable  alternative.  Next,  one  should  consider  whether 
the  group  responses  can  be  aggregated  meaningfully.  If  there  is  doubt 
about  combiining  the  group  responses,  then  one  would  probably  not  want  to 
use  the  Delphi  procedure.  Finally,  one  must  consider  whether  there  are 
any  popular  opinions  to  which  there  may  be  pressure  to  conform.  This  may 
influence  the  committee  chairman  to  pressure  the  group  or  lead  the  group 
in  the  direction  of  a  favored  policy  or  opinion,  even  though  it  may  not 
represent  the  group’s  opinion. 


4. 3. 1.6  Closed  Form  Questionnaire  Technii 


:LLLd 


The  use  of  questionnaires  completed  by  knowledgeable  evaluators  is  a 
technique  for  collecting  information  covering  a  wide  range  of  possible 
functions.  For  our  purposes,  the  information  might  be  for  round  1  of  a 
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Delphi  procedure  for  assigning  risk  measures  of  effectiveness  or  risk 
probability  density  functions.  The  information  may  be  part  of  a  pre¬ 
evaluation  targeting  for  software  areas  needing  more  test  and  evaluation 
scrutiny.  Closed  form  questionnaires  is  a  technique  used  by  AFOTEC  to 
obtain  evaluation  measures  of  software  supportabil ity  (reference  5.1). 


% 


4. 3. 1.6.1  Description. 

The  characterizing  elements  of  a  questionnaire  are  the  response 
scale,  evaluator  sample/characteristics,  and  the  question  organizational 
structure.  Validity  of  a  questionnaire  depends  upon  many  variables,  but 
the  ability  to  repeat  the  questionnaire  used  under  identical  circum¬ 
stances  and  obtain  the  same  results  is  important.  In  order  for  con¬ 
clusions  to  be  reached,  evaluators  must  also  "reasonably"  agree  on  the 
values  to  be  assigned  to  individual  questions. 

Questions  within  only  a  precise  selection  of  responses  are  termed 
closed  form  questions.  For  example,  multiple  choice  questions,  true- 
false  questions,  and  in  general  questions  requiring  a  response  within  a 
lower  and  upper  bound  on  a  linear  scale  are  closed  form  questions.  Essay 
questions  or  questions  allowing  -for  explanation  are  open  form  questions. 
AFOTEC  has  used  a  response  scale  as  indicated  below  for  its  questionnaire 
statements  where  "agreement"  is  good  and  "disagreement"  is  bad. 
Table  4-4  is  an  example  of  one  of  AFOTEC' s  question  statements  and  guide¬ 
lines  on  how  to  interpret  the  scale. 

a)  Completely  agree 

b)  Strongly  agree 

c)  Generally  agree 

d)  Generally  disagree 

e)  Strongly  disagree 

f)  Completely  disagree. 

Normally  questions  are  answered  by  one  or  more  persons  called  evalu¬ 
ators.  The  capability  of  each  evaluator  to  respond  accurately  depends 
upon  the  knowledge  of  the  “valuator  in  the  subject  area  and  upon  the 
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Table  4-4. 

,  Example:  Closed  Form  Question 

!  Question  Number  S  62 

!  QUESTION:  The  number  of  expressions  used  to  control  branching  in  this 

module  is  manageable. 

CHARACTERISTIC:  Simplicity  (size  simplicity). 

Explanations:  The  count  of  control  e  -.pressions  is  closely  related  to  the 
number  of  independent  cycles  in  a  module.  The  more  control  expressions 
there  are  the  more  complex  the  control  logic  tends  to  be. 

EXAMPLES:  The  following  examples  indicate  how  to  count  the  control 

expressions: 

CONTROL  STRUCTURE  STATEMENT  CONTROL  EXPRESSION  COUNT 


Decision  IF  (A.  OR.  B)  GO  TO  10 


IF  (A.  AND.  B)  GO  TO  10 
IF  (C.GT.D)  GO  TO  10 
IF  (A.  AND .B) .  OR.  (C.GT.D)) 
GO  TO  10 
(I)  OF 
A 
8 
C 


A:B 

A:B 

C.GT.D 


CASE 


A:B:C .GT.D 
1=1:1 *2 : 1=3 
(Alternatives) 


END  CASE 


(number  of 
alternatives 
less  one) 


Iteration  DO  10  1*1,  10 
A 

10  CONTINUE 


I.LT.l 

I.GT.10 


GLOSSARY:  Control  Expression:  IF,  CASE,  or  other  decision  control 


expression.  DO,  DO-while,  or  other  iterative  control  expression. 


SPECIAL  RESPONSE  INSTRUCTIONS:  The  following  guidelines  will  anchor  A 
and  f  responses,  but  iri  fairly  subjective  (especially  the  F  anchor). 
The  guidelines  for  the  A  response  is  (sic)  suggested  from  other 
independent  research.  Remember  to  count  al_l_  repetitions  of  the  same 
control  expression  also. 


Answer  A  if  count  <  10 
Answer  F  if  count  >  50 
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clarity  with  which  the  question  is  stated.  If  more  than  one  evaluator  is 
used,  there  is  a  risk  that  there  may  not  be  a  consensus  in  their 
responses.  If  only  one  evaluator  is  used,  natural  evaluation  bias  may 
not  give  an  accurate  result.  Frequently  a  question  is  really  more  than 
one  question  or  the  terms  in  the  question  may  be  misleading  or  undefined. 
This  can  lead  to  evaluation  error  due  to  lack  of  question  reliability. 

Questionnaires  can  be  structured  so  that  groups  and  subgroups  within 
groups  address  particular  functional  parts  of  the  general  subject  area. 
In  this  case,  the  way  in  which  question  responses  are  aggregated  is  of 
importance  as  well  as  the  consistency  of  response  scales  across  subgroups 
and  groups.  Also,  it  is  a  concern  whether  a  group  may  have  too  much  or 
too  little  "natural"  weight  due  to  the  number  of  questions  within  the 
group.  Weights  are  frequently  assigned  by  users  to  groups,  subgroups,  or 
individual  questions  as  a  subjective  level  of  importance  or  as  a  derived 
regression  coefficient  for  use  in  the  aggregation  of  hierarchy  of  evalua¬ 
tion  values.  Sometimes  the  structure  of  the  questionnaire  is  similar  to 
a  decision  tree  where  certain  paths  are  taken  on  the  basis  of  responses 
to  particular  branching  mode  questions.  This  allows  for  a  generic  dis¬ 
crimination  of  what  parts  of  the  subject  are  useful  to  cover  by  the 
questionnaire  as  well  as  exploring  particular  areas  in  more  depth. 


4. 3. 1.6.2  Advantages . 

Closed  form  questionnaires  can  serve  as  valuable  checklists  and 
guidelines  over  a  broad  range  of  a  subject  matter.  Properly  structured 
they  can  quickly  pinpoint  subject  areas  which  are  very  poor  or  very  good, 
and  those,  areas  needing  more  detailed  analysis.  These  type  of  question¬ 
naires  are  quite  flexible  and  can  be  easily  tailored  to  particular 
special  cases.  The  questionnaires  can  form  the  initial  data  point  for 
several  oe  the  other  subjective  (and  even  parametric  objective)  risk 
techniques  including  the  modified  Churchman-Ackoff  technique  and  the 
Delphi  procedure. 
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4. 3. 1.6. 3  Limitations . 


The  limitations  of  closed  form  questionnaires  depends  upon  the 
desired  use  of  the  responses.  The  most  frequent  limitation  is  lack  of 
detail  in  the  response  and  the  subjective  nature  of  the  response.  The 
scale  of  measure  chosen  has  a  great  influence  upon  the  response.  Par¬ 
ticular  linguistic  words  (even  those  dry  ones  such  as  completely  agree) 
can  evoke  evaluator  bias  through  unfavorable  or  favorable  connotations. 

Most  often,  equal  interval  scales  are  selected  and  desired,  but  the 
responses  do  not  fit  an  equal  interval  scale. 

Questions  may  be  very  difficult  to  answer  if  proper  support  tools 
are  not  available.  For  example,  determining  the  extent  of  module  calling 
relationships  without  an  automatically  produced  module  call  cross- 
reference  list  is  very  time-consuming  for  a  large  software  system. 

The  disparity  among  evaluator  responses  may  not  allow  for  meaningful 
conclusions,  especially  for  a  few  number  of  evaluators.  The  sample  of 
evaluators  used  may  not  be  representative  of  the  general  population  of  .  •  r 
evaluators.  Thus,  the  evaluation  may  not  be  repeatable.  Evaluators 
normally  have  bias.  It  usually  requires  automated  capabilities  to 
process  evaluator  responses,  determine  authors,  allow  for  selective 
elimination  of  evaluator  responses,  and  aggregate  questionnaire  response 
values.  Without  automated  support,  the  flexibility  of  using  question¬ 
naires  across  a  range  of  applications  is  limited. 

Since  questionnaires  are  completed  by  evaluators  in  a  manual  manner, 
it  can  take  a  long  time  to  complete  a  questionnaire.  The  utility  of  a 
more  detailed  questionnaire  needs  to  be  carefully  assessed  against  the 
derived  benefit.  Frequently,  it  is  not  possible  to  vary  the  depth  of  the 
questionnaire  and  still  obtain  representative  meaningful  results. 

Most  of  the  limitations  derive  from  lack  of  proper  questionnaire 
design,  evaluator  expertise,  and/or  lack  of  proper  procedures  to  complete 
and  process  the  questionnaire  responses. 
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4. 3. 1.7  8a yes i an  Analysis. 

A  Bayesian  believes  that  it  is  possible  at  any  time  to  express  one's 
state  of  knowledge  about  some  variable  (e.g.,  cost)  in  the  form  of  a 
probability  density  function.  As  additional  experimental  evidence 
becomes  available,  8ayes‘  Theorem  is  used  to  combine  this  evidence  with 
previous  probability  density  functions  in  order  to  obtain  a  new  posterior 
probability  density  function.  For  example,  new  cost  estimates  recently 
obtained  may  be  added  to  a  historical  data  base  of  costs.  The  new  proba¬ 
bility  density  function  represents  the  updated  state  of  knowledge. 


4. 3. 1.7.1  Description. 


Consider  the  Bayesian  analysis  of  p,  an  unknown  parameter  of  a 
postulated  probabilistic  model  of  a  system.  Assume  that  the  experimental 
outcomes  with  the  system  can  be  treated  as  the  values  of  a  random 
variable  X,  the  characteristic  of  interest  (e.g.,  cost).  Based  on  past 
experience  and  all  other  available  information,  the  Bayesian  approach 
begins  with  the  specification  of  a  prior  probability  density  function 
fp(p)»  The  prior  probability  density  function  (PDF)  reflects  the 
analyst's  prior  beliefs  about  the  value  of  the  parameter  p.  The  assumed 
model  specifies  the  probability  density  function  for  the  sample  value  of 
the  characteristic  X,  given  the  value  of  the  parameter  p.  Since  p  is 
being  regarded  as  another  random  variable,  the  PDF  for  the  sample  value 
of  x  with  parameter  p  is  written  as  the  conditional  PDF. 


(Wo* 


Conditional  PDF  for  the  sample  value  of 
characteristic  x.  given  that  the  value  of 
parameter  p  is  equal  to  p 


Each  time  an  experimental  value  of  characteristic  x  is  obtained,  the 
continuous  form  of  B%yes'  Theorem,  listed  here, 
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fp|x(po|xo>  = 


^.p^o-Pq1  .  rx|pfxolpo)  VPq) 


W*olV  W  dp0 


is  used  to  obtain  a  posterior  probability  density  function  fp,  (p0 | xQ ) 


representing  the  analyst's  new  state  of  knowledge  about  the  value  of  the 
parameter  p.  This  posterior  probability  density  function  serves  as  the 
basis  for  any  present  decisions  and  also  as  the  prior  distribution  for 
any  future  experimentation. 

4. 3. 1.7. 2  Advantages . 

In  risk  analysis,  situations  frequently  exist  where  the  analyst  has 
available  both  objective  test  data  and  other  relevant  information  based 
on  the  externalities  of  the  problem.  Often,  due  to  cost  and  time  con¬ 
straints,  there  is  only  a  limited  amount  of  relevant  test  data  available 
by  the  decision  date.  Thus,  other  factors  such  as  previous  test  data, 
engineering  judgment,  experience  with  similar  systems,  etc.,  must  be 
taken  into  consideration.  In  the  context,  Bayesian  statistics  provides 
the  analyst  with  a  tool  for  synthesizing  this  information  into  one  prob¬ 
ability  distribution  which  can  then  be  used  directly  to  estimate  the 
risks  in  question. 

4. 3. 1.7. 3  Limitations . 

Unfortunately,  there  seems  to  be  some  mystique  that  surrounds  any 
application  of  Bayesian  statistics.  This  is  due  in  some  instances  to  a 
disagreement  with  the  Bayesian  philosophy  and  in  others  to  the  lack  of 
true  understanding  of  the  mechanism  of  the  Bayesian  approach.  Further, 
the  mathematics  of  Bayesian  analysis  are  also  fairly  complex.  Advanced 
graduate  training  in  mathematics  or  statistics  is  usually  necessary  to 
implement  this  technique.  Perhaps  one  of  the  most  widely  used  arguments 
against  the  use  of  the  Bayesian  procedure  is  the  apparent  absence  of  a 
rational  basis  for  constructing  a  prior  distribution. 
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4. 3. 1.8  Network  Analysis. 


4. 3. 1.8.1  Description. 
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Most  people  today  are  familiar  with  the  concept  of  a  network  and 
network  modeling.  The  figure  below  is  an  example  of  a  network. 


NETWORK 


In  such  a  network,  each  circle  represents  a  decision  point,  event, 
or  milestone,  and  each  line  represents  an  activity  that  must  be  finished 
to  advance  the  program,  that  consumes  resources,  or  that  takes  time. 
Network  analyses  such  as  PERT  (Program  Evaluation  and  Review  Technique) 
have  been  used  to  manage  schedule  risk  by  establishing  the  shortest 
development  schedules  through  the  network,  by  monitoring  and  projecting 
program  progress,  and  by  funding  and  applying  necessary  resources  for 
maintaining  the  schedule.  Successors  to  PERT  have  estimated  the  minimum 
cost  path  through  the  network. 

Numerous  network  models  and  network  programming  languages  have 
developed  in  recent  years.  Several  of  these  are  quite  sophisticated  (see 
Pritzker  and  Pegden,  reference  5.38).  In  current  network  modeling,  the 
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network  is  defined,  and  for  each  activity,  cost  or  schedule  information 
is  described  in  a  probabilistic  manner.  Then,  by  using  computers  to 
simulate  a  large  number  of  program  completions  through  the  network,  the 
characteristics  of  the  network  can  be  examined. 

During  simulations,  the  events  occur  according  to  the  probabilities 
that  they  were  assigned.  The  probabilities  are  described  using  a  mean 
value  and  variance  to  describe,  say,  the  time  of  completion  for  each 
event  in  the  network.  The  estimates  of  the  mean  and  variance  are  based 
on  subjective  estimates  of  experts.  More  specifically,  probability 
density  functions  are  arrived  at  for  each  event  by: 

a)  Assuming  some  distributional  form  for  the  PDF  for  each 
event. 

b)  Asking  individual  experts  to  determine  for  a  given  param¬ 

eter,  say  cost,  an  optimistic  value,  denoted  “a",  a  pessi¬ 
mistic  value,  denoted  "b",  and  a  most  likely  value, 

denoted  "ml”  for  the  values  for  each  event  in  the  network. 

c)  The  expert  judgements  are  then  combined  into  one  estimate 
of  the  mean  and  variance  of  the  value  for  that  particular 
event. 

Mean  =  (a  +  4ml  +  b)  /  6 
Variance  =  ((b  -  a)/6)  **  2 

Network  model  outputs  typically  include  some  summary  of  the 

thousands  of  simulations  that  are  possible  using  modern  computing  tech¬ 
niques.  These  summaries  can  be  expressed  as  probability  density  func¬ 

tions  or  as  statistical  measures  of  a  particular  value  in  question,  such 
as  cost.  Also,  many  of  the  network  modeling  languages  can  be  used  to 
describe  minimum  paths  through  the  network.  For  instance,  a  least  cost 
path,  on  average  value,  could  be  defined. 


4.3. 1.8.2  Advantages . 

The  obvious  advantage  is  that  networks  can  now  be  evaluated  ju 
thoroughly  using  the  power  of  modern  computers  and  recent  advances  -r  : 
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design  of  simulation  languages.  The  thousands  of  possible  combinations 
of  event  occurrences  are  revealed  using  this  technique.  In  other  words, 
the  tendency  of  humans  to  imagine  scenarios  in  which  only  a  few  factors 
vary  is  circumvented.  A  myriad  of  interactions  within  the  system  are 
examined. 

4. 3. 1.8. 3  Limitations. 

First  and  most  importantly,  the  particular  problem  being  analy2ed 
must  be  a  netwcrk-type  problem.  In  other  words,  network  analysis  is  not 
appropriate  for  all  situations.  Next,  network  analysis  does  involve  a 
degree  of  abstraction  for  the  actual  situation.  The  particular  problems 
must  be  represented  by  decision  points,  events,  etc.  Varying  levels  of 
detail  can  be  used  in  the  definition  of  the  problem  as%  a  network. 
Finally,  the  subjective  estimates  describing  the  events  in  the  network 
(e.g.,  cost)  suffer  from  the  same  problems  as  subjective  estimates  as  a 
whole  do. 


4. 3. 1.9  Decision  Trees. 


4. 3. 1.9.1  Description. 

Decision  trees  are  used  for  the  examination  of  decisions  by  breaking 
them  into  the  sequences  of  supporting  decisions  and  the  resulting 
uncertain  occurrences.  Figure  4-2  is  an  example  of  a  decision  tree. 

In  this  particular  example,  the  left-hand  square  is  the  starting 
point  of  the  sequence  of  decisions.  The  two  circles  to  the  right  of  the 
square  represent  either  of  two  ways  in  which  process  can  move  from  the 
initial  point.  In  this  case,  either  a  tilt-wing  design  can  be  chosen  or 
a  helicopter  can  be  chosen.  From  the  circles,  three  possible  outcomes 
can  occur  on  the  top  branch  (tilt-wing)  or  two  possible  outcomes  can 
occur  on  the  bottom  branch  (helicopter).  The  likelihood  of  each  of  these 
outcomes  is  shown  on  the  appropriate  branch  of  the  decision  tree.  These 
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Figure  4-2.  Examde:  Decision  Tree  Techniques 
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likelihoods,  or  probabilities,  are  assigned  by  some  method  of  eliciting 
subjective  estimates  from  a  panel  of  experts.  This  general  process  of 
designing  *  decision  tree  is  carried  out  until  the  final  consequences  of 
each  possible  branch  of  the  tree  are  shown  on  the  far  right-hand  side. 
In  this  example,  cost  judgnents  are  shown.  Again,  these  judgments  are 
obtained  via  some  subjective  method  of  getting  at  expert  opinions. 


4.3. 1.9.2  Advantages . 

The  decision  tree  method  is  an  easy  and  straightforward  method  of 
modeling  the  possible  outcomes  of  a  situation.  The  method  can  be  imple¬ 
mented  without  extensive  mathematical  training  and  often  provides  a  good 
analysis  of  the  situation. 

4. 3. 1.9. 3  Limitations. 

►A  m 

l  Again,  as  in  the  case  of  network  analysis,  the  decision  tree  is  an 

abstraction  of  the  entire  decision  process.  Thus,  first,  all  of  the 
possible  decisions  should  be  known.  Second,  the  probabilities  assigned 
to  each  branch  of  the  decision  tree  must  be  specified.  Decision  tree 
analysis  presents  no  formal  way  of  estimating  probabilities.  Lastly,  the 
final  outcomes  (e.g.,  cost  values)  must  be  estimated.  Again,  no  explicit 
method  is  specified  in  the  decision  tree  approach.  In  essence,  decision 
tree  analysis  is  a  framework  for  examining  probabilities  and  uncertainty 
once  these  values  are  estimated. 
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4.3.2  Objective  Risk  Techniques. 

The  probability  density  function  can  also  be  estimated  from  objec¬ 
tive  data.  Parametric  models  are  used  for  risk  assessment  where  objec¬ 
tive  data  is  available.  Where  extensive  objective  data  bases  exist, 
accurate  risk  models  have  been  developed.  The  insurance  industry 
inmediately  comes  to  mind.  With  a  great  deal  of  accuracy,  the  auto 
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insurance  business  can  determine  the  probability  of  an  accident  of  vary¬ 
ing  degrees  of  severity. 

4. 3. 2.1  The  Difficulty  of  Making  Objective  Estimates. 

Making  objective  estimates,  i.e.,  obtaining  relationships  among 
variables  based  on  objectively-derived  data,  is  not  necessarily  diffi¬ 
cult.  Whether  the  relationship  correlates  to  the  desired  implication  is 
what  is  difficult  to  determine.  The  risk  relationships  derived  (e.g.,  by 
regression  techniques)  will  involve  only  some  of  the  independent  defining 
variables,  hence  the  derived  model  of  reality  may  not  include  one  or  more 
key  risk  drivers. 

Because  of  the  complexity  involved  with  considering  too  many  risk 
variables,  the  usual  technique  is  to  determine  the  key  risk  drivers. 
This  is  often  a  very  difficult  task,  if  not  impossible.  Also,  each  par¬ 
ticular  case  may  require  somewhat  different  drivers.  Objective  estimates 
depend  upon  the  collection  of  accurate  quantitative  data,  frequently 
based  on  ordered,  equal  interval  scale  (e'.g.,  numeric,  integer).  It  is 
frequently  very  difficult  to  assign  a  quantitative  value  to  a  variable. 

For  example,  assigning  a  numeric  value  between  1  and  10  to  software 
product  quality  is  much  more  difficult  than  obtaining  the  number  of 
alcohol-related  automobile  fatalities  for  each  of  the  fifty  states.  The 
essence  of  the  difficulty  is  that  "software  product  quality"  has  many 
more  defining  characteristics  than  does  alcohol-related  automobile 
fatalities.  The  tendency  in  making  objective  estimates  is  then  to 
decompose  the  "software  product  quality"  into  a  more  definitive  hierarchy 
of  "single-dimension"  characteristics.  The  frequent  net  result  is 
nunerous  characteristics,  nebulous  and  inconsistent  value  scales  (e.g., 
(0,1)  scale,  percent  scale,  (no,  yes)  scale,  1..6  discrete  scale), 
inaccurate  variable  values,  and  even  more  questionable  derived  relation¬ 
ships. 

Of  course,  subjective  risk  estimation  has  many  of  the  same  diffi¬ 
culties  as  objective  risk  estimation.  However,  objective  estimation  may 
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result  in  a  more  dangerous  reliance  upon  "numbers"  rather  than  concept. 
The  tendency  is  to  forget  the  objective-based  theoretical  risk  foundation 
(i.e.,  the  basis  and  meanings  of  the  numbers).  This  may  result  in  a 
blind  application  (perhaps  misapplication)  of  data  to  crunch  out  a 
computer-generated  value  with  little  or  no  evidence  of  how  the  value  was 
derived.  Thus,  it  may  be  known  that  software  supportabil ity  is  risky, 
but  not  why  it  is. 

4. 3. 2. 2  Parametric  Models. 


Parametric  estimating  relationships  are  the  result  of  mathematical 
methods  that  determine  a  relationship  between  some  variable  of  interest 
(e.g.,  cost)  and  measurable  system  characteristics  such  as  code  length, 
maturity,  documentation,  etc.  The  method  makes  use  of  a  statistical 
technique  called  regression  analysis  to  develop  an  equation  to  fit  a  body 
of  data.  The  data  consists,  in  this  example,  of  known  costs  and  the 
associated  system  characteristics  such  as  code  lengths,  maturity 
measures,  documentation  measures,  etc. 

Any  model  is  an  abstraction  of  reality,  by  definition.  A  model  is  a 
way  of  sumnari zing,  representing,  and  expressing  in  a  formal  way  the  com¬ 
plex  relationships  and  interrelationships  of  reality.  In  this  context, 
reality  is  the  software  supportabil ity  problem.  Thus,  it  is  realized 
that  any  parametric  model  will  not  account  for  every  detail  affecting  the 
dependent  variable.  Any  evaluation  of  the  dependent  variable  must  be 

accompanied  by  a  caveat  of  what  is  included  or  excluded  in  the  model. 
Given  that  a  model  is  an  abstraction,  then  the  first  objective  is  to 

identify  the  main  drivers  of  the  dependent  variable.  In  other  words, 

those  components  that  account  for  the  most  variation  in  the  uncertainty 
in  cost,  scheduling,  or  performance  of  software  supportabi 1 ity  will  be 
considered  first  in  the  model  development. 

The  aim  in  developing  a  parametric  model  is  to  first  keep  it  fairly 
parsimonious.  The  rationale  for  parsimony  is  several  fold.  One,  by 

focusing  the  model  only  on  the  key  drivers,  or  independent  variables. 
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undue  complexity  is  avoided.  Increased  model  complexity  may  further 
account  for  the  variation  in  a  given  data  set,  but  the  applicability  of 
complex  models  to  novel  situations  can  be  questionable.  Also,  as  com¬ 
plexity  and  detail  in  a  model  are  added,  then  it  is  implied  that  the 
exactness  of  the  model  improves.  This  may  not  be  the  case.  Further, 
following  Rowe's  (1977)  ideas,  detailed  models  may  contain  more  descrip¬ 
tive  certainty,  yet  there  is  an  increased  measurement  uncertainty.  That 
is,  detailed  concepts  may  be  included  in  a  model,  but  the  measurement  of 
these  concepts  quickly  becomes  problematic.  Finally,  initial  attempts  at 
parametric  modeling  should  not  get  bogged  down  in  detail.  Refining  a 
model  comes  later  after  the  major  pieces  of  a  model  are  in  place.  Thus, 
it  is  the  intent  in  parametric  modeling  to  first  introduce  the  main 
drivers . 

Any  parametric  model  will  not  simply  estimate  some  definitive 
quantity  of  the  dependent  variable.  Instead,  the  model  must  provide,  in 
some  way,  a  set  of  probabilities.  That  is,  some  measure  of  the  variation 
must  be  at  least  appended  to  the  expected  value  of  the  dependent  measure 
(e.g.,  cost).  The  model  must  incorporate  some  notion  of  the  statistical 
uncertainty  of  the  supportabil  ity  expense.  In  this  way,  the  model 
touches  base  with  the  theoretical  basis  of  risk.  Some  estimate  of  a 
probability  density  function  must  be  predicted,  however  crude. 

As  previously  stated,  the  risk  assessment  model  will  be  a  fairly 
simplistic  one.  Perhaps  only  seven  or  eight  risk  factors  will  be  modeled 
to  predict  the  cost,  schedule,  or  performance  measures  of  supportabil ity. 
Factors  such  as  maintenance  and  reliability  have  received  considerable 
attention  in  terms  of  attempting  to  measure  these  concepts.  This  pre¬ 
vious  research  may  be  useful.  Preexisting  parametric-type  relationships 
can  be  directly  incorporated  into  a  model  (given  an  understanding  of 
their  applicability).  More  often  than  not,  however,  well-defined  pieces 
of  a  model  will  not  exist.  For  this  scenario,  the  structural  relation¬ 
ship  of  the  model  must  first  be  determined.  For  instance,  the  cost  of 
supportabi  1  ity  may  be  an  inverse  function  of  the  amount  of  code  documen¬ 
tation.  In  some  cases,  the  driving  factors  may  not  easily  be  measured. 
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Then,  a  proxy  variable  or  a  set  of  proxies  will  be  used.  Where  data 
exists  matching  the  structural  model,  then  parametric  relations  can  be 
developed  via  regression  techniques.  Jackknife  or  bootstrap  methods  can 
be  used  to  incorporate  uncertainty  into  the  model  (see  Efron,  B., 
reference  5.28).  Where  data  is  sparse  or  nonexistent,  then  equations  can 
be  developed  that  are  heuristics  or  "rules  of  thumb."  As  an  example, 
higher  level  computer  languages  are  easier  to  modify  than  assembly 
language  codes.  This  concept  may  be  incorporated  into  a  parametric  model 
as  a  multiplying  factor  of  some  sort.  The  heuristics  can  be  developed  by 
analogy,  from  concepts  published  in  the  literature,  from  intuition,  or 
from  some  reasonable  method  of  obtaining  subjective  estimates. 

Technical  issues  of  the  parametric  modeling  task  are  also  apparent. 
Of  critical  importance  is  the  way  in  which  the  components  or  drivers  of 
the  model  are  combined  together..  Specifically,  if  a  parametric  model 
estimates  probability  density  functions  of  cost  for  only  two  drivers,  say 
maintenance  requirements  and  code  characteristics,  then  it  may  be 
problematic  in  combining  the  estimates  into  a  total  estimate  of  the 
dependent  variable.  The  interdependence  among  drivers  causes  mathemati¬ 
cal  complications  in  building  a  total  probability  distribution  of  the 
dependent  variable.  (See  Worm's  (1981)  paper  for  some  ideas  in  this 
area.)  Another  issue  is  the  distributional  form  of  the  probability 
density  function  describing  the  dependent  variable.  Where  the  prob¬ 
ability  density  function  is  not  completely  and  entirely  determined,  then 
some  distributional  form  is  assumed.  This  assumption  makes  the  modeling 
process  traceable  in  that  only  moments  (e.g.,  mean,  standard  deviation) 
of  the  distribution  need  be  estimated.  From  the  risk  literature 
reviewed,  normal,  beta,  triangular,  Weibul,  and  Rayleigh  distributions 
have  all  been  considered. 


Every  day,  in  our  professional  work  and  our  personal  lives,  each  of 
us  must  make  a  multitude  of  decisions.  Both  major  and  minor,  under 


THE  BOM  CORPORATION 


8DM/A-84-496-TR 


various  conditions  of  uncertainty  and  partial  ignorance.  Decision  theory 
deals  with  the  development  of  methods  and  techniques  that  are  appropriate 
for  making  these  decisions  in  an  optimal  fashion.  In  fact,  statistics 
itself  is  sometimes  described  as  the  science  of  decision-making  under 
uncertainty.  Although  decision  theory  may  not  be  tantamount  to  the 
entire  field  of  statistics,  the  importance  of  decision  theory  has 
steadily  grown  during  the  past  30  years  as  virtually  all  the  classical 
problems  of  statistical  inference,  and  many  new  problems  as  well,  have 
been  formulated  in  decision-theoretic  terms. 

The  mathematical  basis  of  statistical  decision  theory  was  developed 
mainly  be  Abraham  Wald  during  the  1940s.  In  many  respects,  this  theory 
was  an  outgrowth,  and  a  special  case,  of  the  "theory  of  games"  as 
developed  by  von  Neumann  and  others  during  the  1920s  and  1930s.  The 
central  difference  is  that  in  the  theory  of  "zero-sum  two-person  games," 
the  decision-maker  must  act  against  an  intelligent  opponent  whose 
interests  are  diametrically  opposed  to  his  own,  whereas  in  a  statistical 
decision  problem,  there  is  usually  no  such  opponent.  For  this  reason, 
the  theory  of  "minimax  decision  rules,"  which  play  a  central  part  in  the 
theory  of  games,  play  at  best  a  very  minor  part  in  modern  decision 
theory. 

It  is  not  appropriate  to  describe  decision  theory  in  any  depth  in 
this  report.  However,  an  overview  of  some  of  the  more  important  theo¬ 
retical  concepts  will  be  presented  in  the  following  sections.  A  con¬ 
ceptual  example  from  Georgia  Tech  research  on  applying  decision  theory  to 
software  testing  will  illustrate  some  of  these  ideas. 

4. 3. 3.1  Parameters,  Decisions,  and  Consequences. 


Consider  a  problem  in  which  a  decision  maker  (DM)  must  choose  a 
decision  from  some  class  of  available  decisions,  and  suppose  that  the 
consequences  of  this  decision  depend  on  the  unknown  value  9  of  some 
parameter  \.  We  use  the  term  "parameter"  here  in  a  very  general  sense, 
to  represent  any  variable  or  quantity  whose  value  is  unknown  to  the  DM, 
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but  is  relevant  to  his  or  her  decision.  Some  authors  refer  to  X  as  the 
"unknown  state  of  nature"  or  "state  of  the  world."  The  set  ft  of  all 
possible  values  of  X  is  called  the  parameter  space. 

The  set  0  of  all  possible  decisions  d  that  the  DM  might  make  in  the 
given  problem  is  called  the  decision  space. 

For  each  value  of  9  e  ft  and  each  possible  decision  d  e  D,  let  y(9,d) 
denote  the  consequence  to  the  DM  if  he  or  she  chooses  decision  d  when  the 
parameter  has  value  0.  Let  \j>  denote  the  set  of  all  consequences  that 
might  result  from  all  possible  pairings  of  0  and  d.  If  X  has  a  specified 
probability  distribution,  then  the  choice  of  any  particular  decision  d 
will  induce  a  probability  distribution  of  y(X,d)  on  the  set  ^  of  possible 
consequences.  Hence,  the  DM's  choice  among  the  decisions  in  D  is  tanta¬ 
mount  to  a  choice  among  various  probability  distributions  on  the  set 


4. 3. 3. 2  The  Utility  Function. 

The  DM  will  typically  have  preferences  among  the  consequences  in  4>. 
In  some  problems,  these  consequences  might  be  monetary  gains  or  losses; 
in  others  they  might  be  much  more  complicated  and  abstract  quantities. 
In  general,  the  DM's  preferences  among  the  consequences  in  will  result 
in  his  or  her  having  preferences  among  the  different  possible  probability 
distributions  on  <|>.  In  other  words,  if  the  DM  could  have  a  consequence 
from  i|>  generated  by  a  random  process  in  accordance  with  some  specified 
probability  distribution,  he  or  she  would  generally  have  a  preference  as 
to  which  distribution  was  used. 

Now  let  U  denote  a  real-valued  function  on  the  set  i.e.,  a  func¬ 
tion  that  assigns  a  real  number  to  each  consequence  in  4>.  Also,  for  any 
probability  distribution  P  on  the  set  let  E(U  (P)  denote  the  expecta¬ 
tion  of  U  with  respect  to  the  distribution  P.  Then  under  certain  condi¬ 
tions  regarding  the  coherence  of  the  DM's  preferences  among  probability 
distributions,  it  can  be  shown  that  there  exists  such  a  function  U  with 
the  following  property:  for  any  two  distributions,  Pj  and  P^,  is  not 
preferred  to  P^  if  and  only  if  E(UjP^)  ^  E (U  IP2 )  - 
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A  function  U  with  this-  property  is  called  a  utility  function,  and 
the  value  that  U  assigns  to  any  particular  consequence  is  called  the 
utility  of  that  consequence.  The  expected  utility  hypothesis,  as  we  have 
just  described,  states  that  the  DM  will  prefer  a  probability  distribu¬ 
tion  P  for  which  E(U|P)  is  as  large  as  possible.  In  other  words,  the  DM 
will  prefer  a  distribution  for  which  the  expected  utility  of  the  result¬ 
ing  consequence  is  a  maximum. 

It  should  be  noted  that  there  is  more  than  one  utility  function  that 
could  be  used  in  a  given  problem.  If  U  is  a  utility  function,  then  V  = 
aU  +  b,  where  a  and  b  are  constants  (a  >  0,  -  ®  <  b  <  ®)  is  also  a 
utility  function.  The  reason  is  that  for  any  two  distributions  P^  and 
P2,  E (U  | P ^ )  <  E (U  j P2 )  if  an  only  if  E ( V | P^ )  _<  E ( V  | P 2 ) .  Hence,  both  U  and 
V  represent  the  DM's  preferences  equally  well.  In  practice,  this  arbi¬ 
trariness  is  exploited  and  removed  by  choosing  two  particular  con¬ 
sequences  and  assigning  them  the  utilities  0  and  1,  or  0  and  100,  or  some 
other  convenient  pair  of  reference  values. 


4. 3. 3. 3  Components  of  a  Decision  Problem. 


We  now  return  to  the  original  decision  problem.  For  each  value  of 
9  eft  and  each  decision  d  e  D,  let  U(9,d)  denote  the  utility  of  the  con¬ 
sequence  y(9*d).  We  may  think  of  U(9,d)  as  the  utility  of  choosing 
decision  d  when  the  parameter  X  has  the  value  9.  Suppose  that  X  has  a 
specified  probability  distribution  £.  Then  in  accordance  with  the 
expected  utility  hypothesis,  the  DM  will  choose  a  decision  d  for  which 
the  expected  utility  E(U|£,d)  is  a  maximum.  Such  a  decision  is  called  an 
optimal  decision  or  a  Bayes  decision  with  respect  to  the  distribution  £. 

In  many  decision  problems,  it  has  become  standard  to  specify  the 
negative  of  the  utility  function,  rather  than  the  utility  function 


itself,  and  to  call  this  function  the  loss  function.  Thus  the  loss 
L(9,d)  is  the  disutility  to  the  DM  of  choosing  decision  d  when  the  param¬ 
eter  has  the  value  9.  An  optimal  or  Bayes  decision  with  respect  to  the 
distribution  5  will  be  a  decision  d  for  which  the  expected  loss  E(L|£,d) 


is  a  minimum. 
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Thus  the  components  of  a  decision  problem  are  a  parameter  space  a,  a 
decision  space  0,  and  a  loss  function  L(e,d).  For  any  given  distribu¬ 
tion  5  of  x,  the  expected  loss  E(L|£,d)  is  called  the  risk  R(£,d)  of  the 
decision  d.  The  risk  of  the  Bayes  decision,  i.e.,  the  minimum  R  (5)  of 
R(£,d)  over  all  decisions  d  e0,  is  called  the  Bayes  risk. 


4. 3. 3. 4  Subjective  Probability. 


In  some  decision  problems,  the  probability  distribution  £  that  the 
DM  assigns  to  x  will  be  based  on  a  large  amount  of  historical  data  or  on 
theoretical  frequency  considerations.  In  such  problems,  the  distribu¬ 
tion  5  will  be  "objective"  in  the  sense  that  any  other  DM  who  faced  the 
same  problem  would  assign  the  same  distribution.  In  most  decision 
problems,  however,  the  distribution  5  will  be  a  "subjective"  distribution 
that  is  based,  at  least  in  part,  on  the  DM's  personal  information  and 
beliefs  about  what  the  value  of  X  is  likely  to  be. 

The  existence  of  subjective  probabil ities  is  based  on  the  assumption 
that  certain  conditions  are  satisfied  regarding  the  coherence  of  the  DM's 
judgments  about  the  relative  likelihoods  of  various  subsets  of  values  of 
X.  When  these  conditions  are  satisfied,  it  can  be  shown  that  there 
exists  a  unique  probability  distribution  P  on  the  set  $2  that  satisfies 
all  the  mathematical  properties  of  probability  and  has  the  additional 
property  that  for  any  two  subsets  Ac  u  and  3  C  Q,  P(A)  _<  P(B)  if  and 
only  if  the  DM  does  not  believe  that  the  value  of  \  is  more  likely  to  lie 
in  A  than  in  B. 

Some  statisticians  feel  that  there  are  different  types  of  proba¬ 
bility  and  that  subjective  probabilities  are  of  a  different  type  from 
logical,  frequency,  or  physical  probabilities.  On  the  other  hand,  it  can 
be  argued  that  subjective  probability  is  the  only  type  of  probability 
that  can  be  put  on  a  sound  foundation  and  the  only  type  of  probability 
that  exists.  In  this  view,  all  probabilities  are  subjective;  some  are 
more  "objective"  than  others  only  because  larger  groups  of  DM's  would  all 
assign  the  same  values  for  these  probabilities  based  on  their  experience. 
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Together,  the  concepts  of  subjective  probability  and  utility  provide 
a  unified  theory  of  decision  making.  The  OM's  subjective  probabilities 
represent  his  or  her  knowledge  and  beliefs,  and  the  DM's  utilities 
represent  his  or  her  tastes  and  preferences.  The  expert  DM  is  careful  to 
maintain  the  distinction  between  these  concepts,  and  does  not  confuse  the 
value  that  he  or  she  wishes  x  would  have  with  the  value  that  he  or  she 
thinks  x  is  likely  to  have.  In  other  words,  the  DM  does  not  let  utili¬ 
ties  influence  his  or  her  subjective  assignment  of  probabilities,  and 
vice  versa.  The  DM  then  chooses  a  decision  that  maximizes  his  or  her 
subjective  expected  utility  or,  equivalently,  minimizes  his  or  her  sub¬ 
jective  expected  loss. 

4. 3. 3. 5  Decision  Analysis . 

Many  problems  of  decision  making,  such  as  deciding  where  to  locate  a 
new  airport,  are  extremely  complicated,  and  it  is  often  not  immediately 
clear  how  to  apply  the  concepts  of  decision  theory  that  have  just  been 
described.  The  process  of  aiding  the  DM  in  applying  these  concepts  in  a 
particular  problem  is  called  decision  analysis.  In  recent  years  tech¬ 
niques  of  decision  analysis  have  been  developed  which  are  intended  to  aid 
the  DM  in  (a)  identifying  all  the  relevant  dimensions  of  the  parameter  x, 
(b)  specifying  the  spaces  ft  and  D  of  all  possible  parameter  values  9  and 
decisions  d,  and  especially  (c)  specifying  the  DM's  probabilities  and 
util ities . 

Various  procedures  are  available,  including  some  computer  programs, 
for  the  elicitation  of  a  DM's  subjective  probabilities.  A  probability 
distribution  on  fl  must  be  determined  on  the  basis  of  the  DM's  responses 
when  questioned  about  the  relative  likelihoods  of  different  events.  Some 
type  of  fitting  procedure  is  typically  needed  because  few,  if  any, 
persons  exhibit  the  perfect  coherence  necessary  for  the  existence  of  a 
unique  distribution.  Similarly,  procedures  are  available  for  fitting  a 
utility  function  on  the  basis  of  the  DM's  responses  when  questioned  about 
his  or  her  preferences  among  different  probability  distributions  that 
might  yield  a  consequence  from  the  set  i|>. 
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The  standard  problems  of  testing  hypotheses  can  also  be  formulated 
as  decision  problems.  In  fact,  every  test  of  hypothesis  is,  at  least 
theoretically,  a  problem  with  exactly  two  decisions:  accept  the  null 
hypothesis  HQ,  which  we  shall  call  decision  dQ.  and  accept  the  alterna¬ 
tive  hypothesis  (or,  equivalently,  reject  HQ),  which  we  shall  call 
decision  d^. 

Loss  functions  appropriate  to  testing  hypotheses  can  easily  be 
developed.  For  example,  suppose  that  X  is  a  real-valued  parameter  and  it 
is  desired  to  test  the  hypotheses  HQ:x  <  0Q  and  H,:X  >  9Q  where  0Q  is  a 
specified  number.  A  typical  loss  function  for  this  problem  would  have 
the  following  form: 

L(0,do)  =0  for  0  <  eQ, 

L(e,d0)  =  o0(e)  for  0  >  0o, 

L(0,d^)  =  o^(0)  for  0  <  0Q. 

L(e,d^)  a  0  for  9  >  0O> 

where  aQ(0).  is  positive  and  nondecreasing  for  0  >  0Q  and  a^(0)  is  posi¬ 
tive  and  nonincreasing  for  0  <  0  .  The  posterior  PDF  of  X  can  be  calcu¬ 
lated  from  any  specified  prior  PDF.  The  Bayes  test  procedure  would  then 
choose  the  decision  with  the  smaller  posterior  risk. 

4.4  APPLICATION  MODELS. 

The  following  sections  describe  two  actual  models  which  have  been 
proposed  for  risk  assessment  of  software  supportabil ity.  The  first  model 
described  is  currently  being  developed  at  Georgia  Tech.  The  second  model 
was  proposed  by  Fisk  and  Murch  (reference  5.12).  3oth  efforts  to  model 
risk  for  software  supportabi 1 ity  represent  the  only  models  that  appear  to 
exist  for  this  particular  problem. 

Neither  the  Georgia  Tech  model  nor  the  proposed  Fisk/Murch  model 
have  been  identified  with  using  either  subjective  or  objective  data. 
Apparently,  both  models  could  incorporate  either  or  both  types  of  data. 
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Subjective  data  are  those  in  which  no  well-defined  rules  are  used  to 
assign  a  value  (usually  a  number)  to  describe  some  characteristic.  For 
instance,  verbal  estimates  of  code  complexity  are  subjective  data.  On 

the  other  hand,  if  the  complexity  of  a  piece  of  code  is  estimated  as 

"high"  because  there  are  more  than  twenty  modules,  then  the  data  is 
objective.  In  this  instance,  a  well-defined  rule  existed  which  estimated 
complexity  based  on  the  number  of  modules. 

4.4.1  Georgia  Tech  Conceptual  Model:  Software  Testing. 

Georgia  Tech  personnel  (reference  5.39)  are  in  the  conceptual  phase 
of  developing  a  risk  model  for  software  testing.  This  model  is  essen¬ 
tially  a  top  down  approach  based  upon  decision  theory.  The  briefing 
slides  of  reference  5.39  are  not  intended  to  be  an  in-depth  analysis  and 
any  conclusions  are  premature  at  this  time,  but  the  top  view  of  the 

model  does  illustrate  some  aspects  of  decision  theory  such  as  the 

"utility"  function. 

4. 4. 1.1  Description. 

The  basic  information  desired  from  a  risk  model  on  software  testing 
is  the  selection  of  tests  based  upon  optimization  of  residual  risk,  and 
the  determination  of  when  it  is  more  costly  to  continue  testing  than  the 
residual  risk  warrants. 

The  tester  is  presented  with  a  set  of  possible  tests  (T . :  i  =  l..n) 

and  a  set  of  test  strategies  (A^  i  =  l..m)  where  each  A.  may  be  one  or 

more  test,  T.  in  some  test  seguence  based  upon  a  desired  strategy  (e.g., 

highest  reliability  possible).  When  adopting  any  particular  test 

strategy,  a  set  of  consequence  events  can  be  observed  (e.g.,  hardware 

device  x  emits  incorrect  data  value  y).  This  set  of  possible  observed 

events  while  the  system  is  operating  is  denoted  (S . :  i  =  l,e).  For  each 

possible  pair  (Ar  Sj)  a  utility  (tester's)  value  is  determined  and  a 

regret  (in  not  applying  another  strategy)  value  r  is  determined.  The 
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goal  is  to  rank  the  tester's  choices  with  respect  to  the  U^-s,  r^s,  and 
various  optimality  criteria  determined  by  the  test  strategy. 

The  following  test  policies  are  illustrated: 

a)  Policy  1.  High  cost  test  should  be  applied  to  reduce  risk 
of  critical  system  error. 

b)  Policy  2.  Apply  least  cost  test  and  justify  incremental 
costs  in  future  tests. 

c)  Policy  3.  Same  as  Policy  2  for  cases  where  cost  of  test 
may  not  be  linearly  ordered  (i.e.,  parallel  paths  may  be 
alternative  choices). 

Ruby's  Theory  of  Test  Utility  is  briefly  presented.  It  is  based 
upon  test  definition  dependencies:  number  of  variables,  number  of 

domains,  structural  complexity.  The  measure  of  test  cost  is  based  upon 
the  difference  in  the  test  definition  dependencies  from  one  test  to 
another.  The  utility  function  for  a  given  test  is  then  the  product  of 
this  "difference1'  and  the  cost  incurred  due  to  any  remaining  errors  not 
determined  by  the  given  test.  Of  course.,  all  these  dependencies,  dif¬ 
ferences,  and  potential  cost  incurred  by  residual  errors,  are  usually 
very  difficult  to  measure  or  estimate.  Some  possible  methods  of  deter¬ 
mining  the  utility  function  are  illustrated  in  probability  linguistics 
and  possible  test  strategies  described. 

Some  axiomatics  regarding  the  software  test  risk  assessment  are 
given: 


a)  Completely  order  test  strategies 

b)  Isolate  optimal  set. 

Some  guidance  and  terminology  is  presented  for  doing  this.  To  quantify 
the  risk,  it  is  suggested  that  utility  functions  be  derived  using  a  con¬ 
jecture  by  Ruby  concerning  a  simplification  of  the  utility  function 
equation,  that  the  differences  in  the  derived  utility  values  be  dem¬ 
onstrated  to  be  a  measure  of  risk,  and  that  the  preference  relations  for 
testing  be  classified. 


IV-51 


'"Iv 


sj 


J 


■‘M 


V.V.n'.n. 


i  »t  »’«  a'*  •'  i.t’),!  (  i 


At^1  >|T  Sriai‘MJ 


THE  BDM  CORPORATION 


BDM/A-84-496-TR 


4. 4. 1.2  Advantages,  Limitations. 

The  information  in  reference  5.39  is  at  best  sketchy  and  the  inter¬ 
pretation  applied  in  this  report  is  probably  not  totally  accurate.  The 
suggested  model  needs  a  great  deal  of  refinement  and  a  more  practical 
application  example,  even  to  be  reasonably  understood.  However,  it  does 
appear  that  this  type  of  model,  if  it  existed  in  some  reasonably 
validated  form,  would  have  an  importance  for  an  AFOTEC  RAMSS.  The  test 
completeness  index  described  in  reference  5.12  is  dependent  upon  a  test 
strategy.  Knowing  the  various  risks  associated  with  the  test  strategies 
would  first  allow  AFOTEC  to  better  utilize  test  resources  against 
expected  risk  reduction  and  second  provide  test  completeness  risk  MOEs 
of  more  substantial  meaning  than  the  index  suggested  in  reference  5.12. 

4.4.2  Proposed  Fisk/Murch  Model. 


AFOTEC  prepared  a  proposal 'for  computer  resources  risk  assessment 
during  OT&E  (reference  5.12)  as  justification  for  this  feasibility  study. 
This  effort  was  the  only  literature  reviewed  which  directly  addressed  the 
integration  of  risk  assessment  and  software  supportability.  The  proposal 
is  preliminary  and  is  not  officially  sanctioned  by  AFOTEC.  It  was  not 
meant  to  be  classified  as  a  methodology  or  technique,  but  was  simply 
meant  to  be  used  as  a  guide  to  further  study.  Its  importance  derives 
from  the  practical  view  presented  of  evaluating  and  reporting  software 
user  and  supporter  risks  associated  with  acceptance  of  computer 
resources,  especially  software. 

4.4.2. 1  Description. 

This  proposal  presents  a  framework  for  software  risk  assessment. 
This  framework  integrates  aspects  of  current  AFOTEC  developed  methodol¬ 
ogies  for  evaluating  computer  resources  as  part  of  OT&E  activities 
without  restricting  the  possibility  of  including  other  methodologies. 
The  structure  of  this  framework  is  shown  in  figure  4-3. 
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A'. 

The  technique  of  closed  form  questionnaires  (section  4. 3. 1.6)  is  ‘vs 
used  to  arrive  at  software  supportabi 1 ity  risk  "quality  factor"  ratings, 
and  software  operational  risk  "quality  factor"  ratings.  These  ratings 
are  then  normalized  to  a  value  between  0  and  1.  The  independent  quality 
factor  ri sk  is  simply  defined  to  be  one  minus  the  normalized  rating. 

Thus,  the  higher  the  evaluated  quality  factor  value,  the  lower  the 
quality  factor  operational  or  support  risk  and  vice  versa. 

Because  quality  factors  are  not  independent  and  usually  not  of  equal 
importance,  this  risk  assessment  technique  allows  for  a  factor  influence 
matrix  and  a  quality  factor  relative  importance  weight.  The  analytical 
procedure  is  illustrated  in  figure  4-4.  The  procedure  is  applied  for 
user  and  for  supporter  (the  ri  sk  agents  in  this  method).  The  resulting 
risk  values  lie  between  0  and  1  for  user  and  supporter.  The  proposed 
evaluation  criteria  for  "low,"  "medium,"  and  "high"  risk,  and  a  suggested 
matrix  form  representing  the  results,  is  shown  in  figure  4-5. 

An  example  using  the  test  factors  of  figure  4-3  for  user  and  sup-. 
porter  and  hypothesized  values  from  an  AFOTEC  evaluation  is  presented  to 
illustrate  the  analytical  procedure  and  the  resulting  risk  matrix.  A 
condensed  version  of  this  is  shown  in  figure  4-6. 

4.4. 2. 2  Advantages. 

The  advantages  of  the  proposed  Fisk/Murch  model  are  primarily  due  to 
its  direct  applicability  and  simplicity.  This  model  can  easily  be 
applied  within  AFOTEC  evaluation  constraints  (resources,  time).  It  is 
generic  in  that  other  quality  factors  can  be  easily  added  or  the  current 
ones  modified.  The  model  is  not  dependent  on  how  the  risk  MOE  is 
actually  computed.  Thus  different  algorithms  than  the  ones  proposed 
could  be  used.  Or  perhaps  one  of  the  subjective  or  objective  techniques 
discussed  in  sections  4.3.1  or  4.3.2  could  be  used  in  combination  with 
the  current  AFOTEC  closed  form  questionnaire  evaluations  to  estimate  the 
risk  MOE  in  a  more  probabilistic-based  manner. 

The  model  does  provide  a  quick  pointer  hierarchy  to  potential 
problem  (risky)  areas,  from  the  decision  maker  risk  matrix  to  the  user/ 
supporter  risk  agent,  to  the  risk  agent  quality  factors,  to  the  quality 
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Figure  4-4.  Proposed  Fisk/Murch  Risk  Assessment 
Analytical  Procedure 
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Figure  4-6.  Example:  Proposed  Fisk/Murch  Model 


ANALYTIC  PROCEDURE 
(APPLY  INFLUENCE  MATRIX) 


Example;  Proposed  tvsk/Murch  Model  (Concluded) 
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test  factors,  and  eventually  to  the  individual  test  factor  characteris¬ 
tics.  This  is  very  comforting  in  as  subjective  an  area  as  software 
evaluation  to  be  able  to  point  to  lower  level  details  to  support  evalua¬ 
tion  conclusions. 


4. 4. 2. 3  Limitations  . 


The  most  severe  limitation  of  the  proposed  Fisk/Murch  model  is  its 
lack  of  a  theoretical  risk  foundation.  The  risk  MOEs  have  no  direct 
connection  to  "probability  of  a  negative  outcome."  The  reason  for  this 
is  that  the  model  is  not  connected  to  actual  support  activity  and  con¬ 
sequences  other  than  the  quality  factors  which  are  supposed  to  represent 
such  activity  and  consequences.  The  use  of  subjective  weights  as 
importance  factors  has  the  usual  limitation  and  constraint,  i.e.,  the 
weights  are  based  on  "gut  feel."  It  is  very  easy  to  change  a  compuced 
risk  by  one  category  (LO-MED,  MED-HI)  by  manipulating  the  weight. 

The  specific  framework  does  include  the  concept  of  user  and  sup¬ 
porter  as  risk  agents.  The  use  of  the  influence  matrix  recognizes  the 
potential  dependent  interaction  of  what  one  would  like  to  design  as 
independent  quality  factors.  However,  there  is  no  recognition  till  the 
final  integration  of  risk  into  the  decision-maker  matrix,  that  the  user 
and  supporter  have  interdependencies.  For  example,  the  turn-around  time 
required  by  the  user  as  part  of  the  "emergency  maintenance  request"  will 
dictate  the  evaluation  results  for  a  software  support  facility  (a  sup¬ 
porter  factor).  Thus,  as  the  user  risk  from  support  service  decreases 
(i.e.,  turnaround  time  is  reduced),  the  corresponding  supporter  risk 
increases.  The  manner  in  which  influence  matrix  values  are  computed  is 
also  highly  questionable  from  a  feasibility  viewpoint.  It  is  probably 
impossible  to  determine  an  accurate  numerical  value  for  the  impact  of 
test  completeness  upon  maturity  or  vice  versa.  The  non-symetric  nature 
of  this  relationship  was  alluded  to  by  the  form  of  the  user  influence 
matrix  in  figure  4-6,  but  the  proposal  did  not  make  it  clear. 

The  combination  of  scales  with  vastly  different  meanings  (e.g.  test 
completeness  and  operator-machine  interface)  is  a  severe  limitation  for 


THE  BDM  CORPORATION 


BDM/A-84-496-TR 


the  user  quality  factors.  Note  that  the  scales  for  the  supporter  quality 
factors  are  the  same.  Also,  there  are  some  minor  technical  inconsis¬ 
tencies  with  the  normalization  of  values.  For  example  the  MOE  risk  for 
source  listings  is  (1  -  Sl/6),  but  the  source  listing  values  (SL)  range 
from  1  to  6,  so  a  risk  value  of  1  is  impossible.  Perhaps  a  better  norm¬ 
alization  form  would  be  (1  -  ( SL- 1 ) / 5 ) . 
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APPENDIX  A 
ACRONYMS 


AFOTEC 


CRISP 


DPESO 

OSARC 

OTIC 


Association  for  Computing  Machinery 
Automatic  Data  Processing 
Automatic  Data  Processing  Equipment 
Automatic  (Automated)  Data  Processing  System 
Automated  Data  Processing  System 
Automated  Data  System 

Air  Force  Operational  Test  and  Evaluation  Center 

Air  Force  Regulation 

Ada  Programming  Support  Environment 

Configuration  Control  Board 

Critical  Design  Review 

Cost  Estimating  Relationship 

Configuration  Item 

Configuration  Management 

Configuration  Management  Plan 

Configuration  Management  System 

Computer  Program  Configuration  Item 

Central  Processing  Unit 

Computer  Resources  Integrated  Support  Plan 

Computer  System  Security 

Designated  Approving  Authority 
Data  Base  Change  Request 
Decision  Coordinating  Papers 
Data  Item  Description 
Decision  Maker 

Data  Processing  Installation 
Department  of  Defense 
OoD  Product  Engineering  Services  Office 
Defense  System  Acquisition  Review  Council 
Defense  Technical  Information  Center 
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ECS 

Embedded  Computer  System 

ELC 

Emergency  Low  Complexity 

FCA 

Functional  Configuration  Audit 

FIPS 

Federal  Information  Processing  Standard 

GAO 

Government  Accounting  Office 

ICA 

Independent  Cost  Analysis 

IOT&E 

Initial  Operational  Test  and  Evaluation 

I  SR 

Independent  Schedule  Review 

IV&V 

Indpendent  Verification  and  Validation 

MAJCOM 

Major  Command 

MOE 

Measure  of  Effectiveness 

NBS 

National  Bureau  of  Standards 

NTIS 

National  Technical  Information  Service 

OMB 

Office  of  Management  and  Budget 

OPR 

Office  of  Primary  Responsibil  ity 

OT&E 

Operational  Test  and  Evaluation 

PCA 

Physical  Configuration  Audit 

PDF 

Probability  Density  Function 

PDR 

Preliminary  Design  Review 

PERT 

Program  Evaluation  and  Review  Technique 

PMD 

Program  Management  Directive 

PMP 

Program  Management  Plan 

PRR 

Program  Readiness  Review 

PVR 

Product  Verification  Review 

QA 

Quality  Assurance 

RA 

Risk  Assessment 

RADC 

Rome  Air  Development  Center 

RAMSS 

Risk  Assessment  Model  for 'Software  Supportabil ity 

SA 

Security  Audit 

SAB 

Scientific  Advisory  Board 

SCP 

System  Concept  Papers 

THE  BOM  CORPORATION 


SDR 

System  Design  Requirement 

SOA 

Special  Operating  Agency 

SS 

Software  Supportabil ity 

SSA 

Source  Selection  Authority 

SSAC 

Source  Selection  Advisory  Council 

SSF 

Software  Support  Facility 

SVR 

System  Validation  Review 

SWM 

Software  Maintainability 
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APPENDIX  B 
GLOSSARY  OF  TERMS 


B.l  INTRODUCTION, 


The  glossary  of  terms  for  the  Analysis  of  Software  Supportabi 1 ity 
Risk  Assessment  models  will  vary  as  the  project  progresses.  Refer  to 
BDM/A-84-322-TR,  Final  to  be  dated  September  28,  1984,  for  the  complete 
glossary  of  terms. 

Some  terms  have  more  than  one  description;  when  this  is  the  case, 
the  descriptions  either: 

a)  Are  signif icantly  different  between  sources  (though  the 
effective  meaning  may  be  not  much  different). 

b)  Are  used  differently  (different  users  or  technical  langu¬ 
age). 

c)  May  be  found  within  the  context  of  a  different  source. 

d)  Have  real  differences  in  meaning. 

Both  DoO  and  non-DoD  (e.g.,  FIPS  PUBs,  NBS  Special  Publications)  sources 
are  used.  The  non-DoD  sources  and  terms  are  not  mandated  for  our  use, 
but  are  rather  included  for  breadth  of  understanding,  for  those  relevant 
terms  comnonly  used  within  the  non-DoD  governmental  and/or  private 
sectors . 

The  source  of  each  description  is  indicated  by  a  symbol  in  paren¬ 
thesis  before  that  source’s  term  description: 

TERM: 

(SYMB0lia) 

Description^  ^ . . . 

(SYMB0L1  2) 

Description^  2"  • 


TERM 


( SYMBOL 1>n) 
Description^  R. 


TERM. 
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The  symbols  used  and  corresponding  sources  are: 

(AF0TECP1)  AFOTECP  800-2,  Volume  1,  10  Nov  82,  "Software  Test 

Manager's  Guide." 

(AF0TECP3)  AFOTECP  800-2,  Volume  III,  1  Jan  84,  "Software  Maintain¬ 
ability  Evaluator's  Guide." 

(AFR800-14)  Air  Force  Regulation  800-14,  Volume  I,  "Management  of  Com¬ 
puter  Resources  in  Systems,"  12  Sep  75. 

(AFR300-15)  Air  Force  Regulation  300-15,  "Automated  Data  System 
Project  Management,"  Jan  78. 

(AFOTECP 5)  AFOTECP  800-2,  Volume  5,  25  Jul  83,  "Software  Support 

Facility  Evaluation — User's  Guide." 

(ROWE)  Rowe,  William,  An  Anatomy  of  Risk,  John  Wiley,  1977. 

(LATHROP)  Lathrop,  Frank,  "Alternative  Methods  for  Risk  Analysis:  A 
Feasibility  Study,"  Air  Force  Computer  Security  Program 
Office,  1  Sep  81. 

(AFR205X)  Air  Force  Regulation  205-16,  "Automatic  Data  Processing 

(AOP)  Security  Policy,  Procedures  and  Responsibilities, 

1  Aug  84. 

(CURRENT)  Current  document  definition. 
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B.2  GLOSSARY  OF  TERMS  FOR  THE  ANALYSIS  FOR  DETERMINING  FEASIBILITY 
OF  DEVELOPING  AND  IMPLEMENTING  A  RISK  ASSESSMENT  MODEL  FOR 
SOFTWARE  SUPPORTABILITY . 

Accuracy 

(ROWE) 

The  quality  of  being  free  from  error.  The  degree  of  accuracy  is  a 
measure  of  the  uncertainty  in  identifying  the  true  measure  of  a 
quantity  at  the  level  of  precision  of  the  scale  used  for  the  quan¬ 
tity. 

Algorithm 

(AFOTECP3) 

A  prescribed  set  of  well-defined  rules  or  processes  for  the  solution 
of  a  problem  in  a  finite  number  of  steps. 

Allocated  Baseline 

(AFR300-15) 

The  initial  approved  allocated  configuration  identification  estab¬ 
lished  at  end  of  the  definition  phase. 

Alternative 

(ROWE ) 

One  member  of  a  set  of  options  associated  with  a  decision,  the 
decision  being  limited  to  a  choice  of  one  and  only  one. 

Application  Functions 

(AF0TECP3) 

Any  functions  which  provide  specific  operational  (mission)  computa¬ 
tions  . 

Application  Software 
(AF0TECP5) 

The  software  written  by  software  support  personnel,  or  purchased 
from  a  contractor,  used  directly  in  supporting  ECSs.  It  is  normally 
used  for  simulation,  testing,  and  ECS  code  development. 

Application  Software  (functional) 

(AFR205X) 

Those  routines  and  programs  designed  by  or  for  automatic  data 
processing  system  users  and  customers  to  complete  specific,  mission- 
oriented  task,  jobs,  or  functions,  using  available  automated  data 
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processing  equipment  and  basic  software.  Application  Software  may 
be  either  general  purpose  packages,  such  as  demand  deposit  account¬ 
ing,  payroll,  machine  tool  control ,  etc.,  or  specific  application 
programs  tailored  to  complete  a  single  or  limited  number  of  user 
functions  (for  example,  base  level  personnel,  depot  maintenance, 
aircraft,  missile  or  satellite  tracking,  command  and  control,  etc.). 

Except  for  general  purpose  packages  that  are  acquired  directly  from 
software  vendors  or  from  the  original  equipment  manufacturers,  this 
type  of  software  is  generally  developed  by  the  user,  either  with  in- 
house  resources  or  through  contract  services. 

Approval  to  Operate 

(AFR205X) 

Represents  concurrence  by  the  designated  approving  authority  (DAA) 
that  a  satisfactory  level  of  security  (that  is,  minimum  requirements 
are  met  and  an  acceptable  level  of  risk  exists)  has  been  provided, 
and  authorizes  the  operation  of  an  automated  data  processing  system 
(ADPS)  or  network  at  an  automatic  data  processing  facility  (ADPF). 
Approval  results  from  an  analysis  of  the  ADPF,  ADPS,  and  automatic 
data  system  (ADS)  certifications  and  the  operational  environment  of 
the  automatic  data  processing  (AOP)  entity  by  the  DAA. 

Attributes 

(AF0TECP3) 

Type,  units,  range,  description,  etc.,  as  appropriate. 

Automated  Decisionmaking  System 
(AFR205X ) 

Those  computer  applications  which  issue  checks,  requisition  sup¬ 
plies,  or  perform  similar  functions  based  on  programmed  criteria, 
with  little  human  intervention. 

Automated  Software  Development  Tool 

(AF0TECP5) 

A  component  of  System  Software  that  assists  in  the  design,  imple¬ 
mentation,  documentation,  and  verification  of  ECS  software. 

Automatic  Data  Processing  Facility  (ADPF) 

(AFR205X) 

The  physical  resources,  including  structures  or  parts  of  structures, 
which  house  and  support  data  processing  capabilities.  For  each  com¬ 
puter  facility  designated  as  a  data  processing  installation  (DPI, 
reference  AFR  300-6),  the  ADPF  is  the  DPI.  For  small  computers, 
stand-alone  systems,  and  word  processing  equipment,  the  ADPF  is  the 
physical  area  in  which  the  computer  is  used.  v\ 
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Automatic  Oata  Processing  Resources 
(AFR205X) 

The  totality  of  automatic  data  processing  equipment,  software,  data, 
computer  time,  computer  programs,  automatic  data  processing  (ADP) 
contractual  services,  ADP  personnel,  and  supplies. 

Availability 

(AFR800-14) 

A  measure  of  the  degree  to  which  an  item  is  in  the  operable  and 
commitable  state  at  the  start  of  the  mission,  when  the  mission  is 
called  for  at  an  unknown  (random)  point  in  time.  (MIL-STD-721) 

(AF0TECP5) 

The  probability  that  a-  system  is  operating  satisfactorily  at  any 
point  in  time  when  used  under  stated  conditions. 

Basel ine 

(AFR300-15) 

A  configuration  identification  document  or  set  of  such  documents 
formally  designated  and  fixed  at  a  specific  time  during  a  CPCI's 
life  cycle.  Baselines,  plus  approved  changes  to  those  baselines 
constitute  the  current  conf iguration  identif ication . 

(ROWE) 

A  known  reference  used  as  a  guide  for  further  development  activi¬ 
ties. 

Bayesian  Statistics 
(ROWE) 

"Bayes  rule"  (Thomas  3ayes,  a  nineteenth  century  English  mathematic¬ 
ian  and  clergyman)  states  that  the  probability  that  both  of  two 
events  will  occur  is  the  probability  of  the  first  multiplied  by  the 
probability  that  if  the  first  has  occurred,  the  second  will  also 
occur.  8ayesian  statistics  is  a  way  of  making  quantity  of  informa¬ 
tion  substitute  for  quality  of  information.  There  are  two  kinds  of 
probability:  the  classical  type  derived  from  empirical  information, 
and  subjective  probability.  Bayesian  statistics  is  based  on  these 
"subjective  probabilities."  It  involves  the  joint  probability  of  A 
and  B.  The  probability  of  the  second  event  occurring  if  the  first 
has  occurred  is  called  the  conditional  probability  of  the  second, 
given  the  first.  Stated  another  way,  the  probability  of  any  event 
P(A)  is  always  positive  but  never  greater  than  1.  Symbolically,  Of 
P(A)  <  1.  If  P(A)  =  0,  the  occurrence  of  the  event  B  is  considered 
impossible.  If  P(A)  =  1,  the  occurrence  of  the  event  B  is  con¬ 
sidered  to  occur  with  P(B). 
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Benefit 

(ROWE) 

a)  An  axiological  concept  representing  anything  received  that  causes 
a  net  improvement  to  accrue  to  the  recipient. 

b)  A  result  of  a  specific  action  that  constitutes  an  increase  in  the 
production  possibilities  or  welfare  level  of  society. 

Benefit-Cost  Ratio 

(ROWE) 

The  ratio  of  total  social  benefit  to  total  social  costs  related  to  a 
specific  activity. 

Capabi 1 ity 

(ROWE) 

A  measure  of  the  degree  to  which  a  system  is  able  to  satisfy  its 
performance  objectives. 

Cardinal  (interval)  Scale 

(ROWE) 

A  continuous  scale  between  two  end  points,  neither  of  which  is 
necessarily  fixed. 

Computer  Program 

(AFR800-14) 

A  series  of  instructions  or  statements  in  a  form  acceptable  to  an 
electronic  computer,  designed  to  cause  the  computer  to  execute  an 
operation  or  operations. 

Computer  Resources 

(AFR800-14) 

The  totality  of  computer  equipment,  computer  programs,  associated 
documentation,  contractual  services,  personnel  and  supplies. 

Configuration  Control 

(AFR3GO-15) 

The  systematic  evaluation,  coordination,  approval  or  disapproval, 
and  implementation  of  approved  changes  in  the  configuration  of  a 
CPCI  after  formal  establishment  of  its  configuration  identification. 

Conf iguration  Item  (Cl) 

(AFR300-15) 

An  item  of  AOPE  that  is  designated  for  configuration  management. 
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(AFR800-14) 

An  aggregation  of  equipment/software,  or  any  of  its  discrete  por¬ 
tions,  which  satisfies  an  end  use  function  and  is  designated  by  the 
Government  for  configuration  management.  CIs  may  vary  widely  in 
complexity,  size  and  type,  from  an  aircraft  or  electronic  system  to 
a  test  meter  or  round  of  ammunition.  During  development  and  initial 
production,  CIs  are  only  those  specification  items  that  are  refer¬ 
enced  directly  in  a  contract  (or-  an  equivalent  in-house  agreement). 
During  the  operation  and  maintenance  period,  any  reparable  item 
designated  for  separate  procurement  is  a  configuration  item. 
(AFR  65-3) 

Configuration  Management  (CM) 

(AFR300-15) 

A  management  discipline  that  applies  technical  and  administrative 
direction  and  surveillance  to: 

(1)  Identify  and  document  the  functional  and  physical  charac¬ 
teristics  of  a  configuration  item. 

(2)  Control  changes  to  those  characteristics. 

(3)  Record  and  report  configuration  status. 

Configuration  Management  Plan  (CMP) 

(AFR300-15) 

A  document  which  describes  project  responsibilities  and  procedures 
for  implementing  CM. 

Configuration  Management  System  (CMS) 

(AF0TECP5) 

A  system  applying  technical  and  administrative  direction  and  sur¬ 
veillance  to  identify  and  document  the  functional  and  physical 
characteristics  of  a  configuration  item;  to  control  changes  to  those 
characteristics  and  to  record  and  report  change  Processing  and 
implementation  status. 

Consequence  Value 

(ROWE) 

The  importance  a  risk  agent  subjectively  attaches  to  the  undesir¬ 
ability  of  a  specific  risk  consequence. 

Consensus 

(ROWE) 

Group  solidarity  in  sentiment  and  bel ief ..  .general  agreement. 
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Cost  v 

(ROWE) 

A  result  of  a  specific  action  that  constitutes  a  decrease  in  the 
production  possibilities  or  welfare  level  of  society.  Also  see 
Loss. 

Cost-Benefit  Analysis 
(ROWE) 

An  attempt  to  delineate  and  compare  in  terms  of  society  as  a  whole 
the  significant  effects,  both  positive  and  negative,  of  a  specific 
action.  Generally  a  number  of  alternative  actions  are  analyzed 
resulting  in  the  selection  of  the  alternative  that  provides  either 
the  largest  benefit-cost  ratio  (total  benefit/total  cost)  or  one 
with  a  positive  ratio  at  least.  If  an  alternative  results  in  a  net 
benefit  less  than  zero  or  a  benefit-cost  ratio  less  than  1,  it  is 
deemed  socially  inefficient  and  is  not  carried  out. 

Cost-Effectiveness  Analysis 

(ROWE) 

A  term  less  specific  than  cost-benefit  analysis,  usually  meaning  the 
selection  of  the  lowest  cost  alternative  that  achieves  a  predeter¬ 
mined  level  of  benefits.  Alternatively,  the  analysis  and  selection 
of  the  path  that  yields  the  largest  social  benefit  for  a  predeter¬ 
mined  specified  level  of  social  costs. 

Critical  Automatic  Data  Processing  Resources 

(AFR205X) 

Those  resources  that  must  be  protected  because  their  compromise, 
alternation,  destruction,  loss,  or  failure  to  meet  objectives  will 
jeopardize  the  accomplishment  of  an  Air  Force,  Air  Force  subelement, 
or  other  service  mission  or  the  accomplishment  of  DoD  life  support 
functions . 

Critical  Design  Review  (CDR) 

(AFR300-15) 

A  formal  review  conducted  during  the  development  phase  before  trans¬ 
lating  logic,  and  algorithms  to  coded  instructions. 

Critical  Issues 

(AF0TECP1) 

Those  aspects  of  a  system's  capability,  either  operational,  techni¬ 
cal,  or  other,  that  must  be  questioned  before  a  system's  overall 
worth  can  be  estimated  and  that  are  of  primary  importance  to  the 
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decision  authority  in  reaching  a  decision  to  allow  the  system  to 
advance  into  the  next  acquisition  phase.  (DoD  Directive  5000.3). 

Data  Item  Description 

(AFR800-14) 

A  form  which  specifies  an  item  of  data  required  to  be  furnished  by  a 
contractor.  This  form  specifically  defines  the  content,  preparation 
instructions ,  format  and  intended  use  of  each  data  product. 


instructions , 
(AFR  310-1) 

Decision  Analysis 


product . 


(ROWE) 

A  methodology  of  decomposition  of  the  decision-making  process  into 
parts,  whereby  the  appropriate  data  can  be  associated  with  the 
parts,  to  provide  a  rational  basis  for  decision  making. 

Decision  Making 

(ROWE) 

A  dynamic  process  of  interaction,  involving  information  and  judgment 
among  participants  who  determine  a  particular  policy  choice. 
Decision  models  are  either  models  of  the  decision-making  process 
itself,  or  analytical  models  (e.g.,  decision  trees,  decision  matri¬ 
ces)  used  as  aids  in  arriving  at  the  decisions.  Decision  theories 
usually  are  in  relation  to  the  process  itself. 

Decision  Matrices 

(ROWE) 

Matrices  whose  elements  exhibit  quantitative  relationships  (cardinal 
or  ordinal)  among  sets  of  factors  coming  into  play  in  the  decision¬ 
making  process. 

Decision  Tree 

(ROWE) 

A  device  used  to  portray  alternative  courses  of  action  and  relate 
them  to  alternative  decisions  showing  all  consequences  of  the 
decision.  The  tree  represents  alternative  courses  or  series  of 
actions  related  to  a  previous  decision. 

Oecisive  Decision  Conditions 


(ROWE) 

Conditions  in  which  the  preference  between  values  on  a  utility  scale 
is  clearly  discernible  because  ranges  of  uncertainty  of  the  two 
values  do  not  overlap  (in  the  case  of  uniform  distributions  of 
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uncertainty)  or  are  below  a  certain  error  level  (for  normal  distri¬ 
butions  of  uncertainty). 

Degree  of  Uncertainty 

(ROWE) 

That  proportion  of  information  about  a  total  system  that  is  unknown 
in  relation  to  the  total  information  about  the  system. 

Delphi  Technique 

(ROWE) 

An  iterative  method  designed  to  produce  a  consensus  by  repeated 
queries  of  an  individual  with  feedback  of  group  responses.  Members 
of  the  group  do  not  interact  directly. 

Descriptive  Uncertainty 

(ROWE) 

The  absence  of  information  about  the  completeness  of  the  description 
of  the  degrees  of  freedom  of  a  system. 

Designated  Approving  Authority 

(AFR205X) 

An  official  designated  to  approve  the  operation  of  automatic  data 
processing  systems  at  the  automatic  data  processing  facilities  under 
his  or  her  jurisdiction  for  storage  of  classified  or  sensitive 
unclassified  information  or  for  critical  processing. 

Devi ation 

(AFR300-15) 

A  written  authorization,  granted  prior  to  the  development  of  a  CPCI, 
to  depart  from  a  particular  performance  or  design  requirement;  a 
specification  for  a  specific  number  of  units;  a  specific  period  of 
time;  or  established  standards. 

Documentation 

(AF0TECP5) 

All  of  the  written  work  describing  operating  and  maintenance  proced¬ 
ures  for  a  system. 

Documentation  Consistency 

(AF0TECP5) 

A  measure  of  the  consistency  in  the  information  provided  in  support 
system  documentation. 
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Documentation  Oescriptiveness 
(AF0TECP5) 

A  measure  of  the  descriptiveness  of  the  information  provided  in 

support  system  documentation. 

Documentation  Modularity 

(AF0TECP5) 

A  measure  of  the  modular  organization  of  information  provided  in 

support  system  documentation. 

Documentation  Simplicity 

(AF0TECP5) 

A  measure  of  the  ease  of  use  and  lack  of  complexity  in  the  informa¬ 
tion  provided  in  computer  system  documentation. 

Embedded  Computer  Resources 

(AF0TECP1) 

Computer  resources  incorporated  as  integral  parts  of,  dedicated  to, 
required  for  direct  support  of,  or  for  the  upgrading  or  modification 
of  major  or  less  than  major  system(s).  (Excludes  ADP  resources  as 
defined  and  administered  under  AFR  300  series.)  (USAF/RD/LE  Policy 
letter,  13  October  1981). 

Embedded  Computer  System  (ECS) 

(AF0TECP1) 

a)  A  computer  that  is  integral  to  an  electromechanical  system  and 
that  has  the  following  key  attributes: 

(1)  Physically  incorporated  into  a  large  system  whose  primary 
function  is  not  data  processing. 

(2)  Integral  to,  or  supportive  of,  a  larger  system  from  a 
design,  procurement,  and  operations  viewpoint. 

(3)  Inputs  include  target  data,  environmental  data,  command 
and  control,  etc. 

(4)  Outputs  include  target  information,  flight  information, 
control  signals,  etc. 

b)  In  general,  an  embedded  computer  system  (ECS)  is  developed, 
acquired,  and  operated  under  decentralized  management.  (DoD  Direc¬ 
tives  5000.1,  5000.2). 

(AF0TECP5) 

A  computer  that  is  integral  to  an  electronic  or  electromechanical 
system  (e.g.,  aircraft,  missile,  spacecraft,  communications  device) 
from  a  design,  procurement,  and  operational  viewpoint. 
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Empirical 

(ROWE) 

Originating  in  or  based  on  observation  or  experience. 

Equitable  Risk 
(ROWE) 

A  risk  agent  receives  direct  benefits  as  a  result  of  exposure  to  a 
risk,  and  the  knowledge  of  the  risk  is  not  purposely  withheld  from 
the  risk  agent. 

Estimation 

(ROWE) 

The  assignment  of  probability  measures  to  a  postulated  future  event. 
Estimator  Uncertainty 
(ROWE) 

Uncertainty  in  measurement  resulting  from  deliberate  use  of  less 
complex  measures  such  as  central  value  estimates  of  dispersion  and 
smoothing  functions  for  time-dependent  parameters. 

Evaluation 

(ROWE) 

Comparison  of  performance  of  an  activity  with  the  objectives  of  the 
activity  and  assignment  of  a  success  measure  to  that  performance. 

Evaluation  Criteria 

(AF0TECP1) 

Standards  by  which  achievement  of  required  operational  effective¬ 
ness/suitability  characteristics  or  resolution  of  technical  or 
operational  issues  may  be  judged.  For  full-scale  development  and 
beyond,  evaluation  criteria  must  include  quantitative  goals  (the 
desired  value)  and  thresholds  (the  value  beyond  which  the  character¬ 
istic  is  unsatisfactory)  whenever  possible.  (DoO  Directive  5000.3). 

Event 

(ROWE) 

A  particular  point  in  time  associated  with  the  beginning  or  comple¬ 
tion  of  an  activity,  and  possibly  accompanied  by  a  statement  of  the 
benefit  or  result  attained  or  to  be  attained  because  of  the  comple¬ 
tion  of  an  activity. 
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Expandability 

(AF0TECP5) 

A  measure  of  the  ease  with  which  the  functional  capability  of  com¬ 
puter  hardware  or  software  may  be  expanded. 

Expected  Value,  Use  Of 

(ROWE) 

Valuation  of  an  uncertain  nunerical  event  by  weighting  all  possible 
events  by  their  probability  of  occurrence  and  averaging. 

Expert  Judgment 

(ROWE) 

Designating  the  relevance  of  opinions  of  persons  well  informed  in  an 
area  for  estimates  (e.g.,  forecasts  of  economic  activity). 

Exposure  (to  risk) 

(ROWE) 

The  condition  of  being  vulnerable  to  some  degree  to  a  particular 
outcome  of  an  activity,  if  that  outcome  occurs. 

Extrapolation/Projection 

(ROWE) 

The  technique  of  estimating  the  future  by  a  continuation  of  past 
trends  without  attempts  to  understand  the  underlying  phenomena. 

Facility 

(AF0TECP5) 

The  physical  plant  and  the  services  it  provides;  specific  examples 
are  physical  space,  electrical  power,  physical  and  electromagnetic 
(TEMPEST)  security,  environmental  control,  fire  safety  provisions, 
and  communications  availability. 

Feasible 

(ROWE) 

That  which  is  possible  to  do,  realistically. 

Feedback 

(ROWE) 

The  return  of  performance  data  to  a  point  permitting  comparison  with 
objective  data,  normally  for  the  purpose  of  improving  performance 
(goal-seeking  feedback),  but  occasionally  to  modify  the  objective 
(goal-changing  feedback). 
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(AF0TECP1) 

a)  Computer  programs  and  data  loaded  in  a  class  of  memory  that 
cannot  be  dynamically  modified  by  the  computer  during  processing. 

b)  Hardware  that  contains  a  computer  program  and  data  that  cannot  be 
changed  in  its  application  environment. 

Note  1.  The  computer  programs  and  data  contained  in  firmware  are 
classified  as  software;  the  circuitry  containing  the  computer  pro¬ 
gram  and  data  is  classified  as  hardware.  (Data  and  Analysis  Center 
for  Software). 

Functional  Configuration  Audit  (FCA) 

(AFR300-15) 

The  formal  examination  of  CPCI  to  verify  that  the  performance  speci¬ 
fied  in  the  SS  has  been  achieved. 

Independent  Verification  and  Validation  (IV&V) 

(AF0TECP1) 

An  independent  assessment  process  structured  to  ensure  that  computer 
programs  fulfill  the  requirements  stated  in  system  and  subsystem 
specifications  and  satisfactorily  perform  the  functions  required  to 
meet  the  user's  and  supporter's  requirements.  IV&V  consists  of 

and  vali da- 


three  essential 
tion: 

(1)  Indepi 


elements:  independence,  verification. 


(1)  Independent.  An  organi zation/ agency  which  is  separate 
from  the  software  development  activity  from  a  contractual 
and  organizational  standpoint. 

(2)  Verification.  The  evaluation  to  determine  whether  the 
products  of  each  step  of  the  computer  program  development 
process  fulfill  all  requirements  levied  by  the  previous 
step. 

(3)  Validation.  The  integration,  testing,  and/or  evaluation 
activities  carried  out  at  the  system/subsystem  level  to 
evaluate  the  developed  computer  program  against  the  system 
specifications  and  the  user's  and  supporter's  require¬ 
ments.  { AFR  88-14) 

Individual  Risk  Evaluation 
(ROWE) 

The  complex  process,  conscious  or  unconscious,  whereby  an  individual 
accepts  a  given  risk. 
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Inequitable  Risk 
(ROWE) 

A  risk  agent  is  exposed  to  a  risk  and  receives  no  direct  benefits 
from  such  exposure,  or  the  knowledge  of  the  risk  is  purposely  with¬ 
held  from  him. 

Interdependence 

(ROWE) 

A  property  shared  by  two  or  more  entities  whenever  the  performance 
of  any  one  affects  the  performance  of  some  or  all  the  rest. 

Interoperabi 1 ity 

(AF0TECP5) 

A  measure  of  the  degree  to  which  computer  hardware  or  software  can 
interface  to  and  operate  with  other  similar  computer  hardware  or 
software. 

Intrinsic  Parameter 
(ROWE) 

A  variable  whose  measurement  is  based  on  the  value  system  of  an 
individual  and  his  perception  of  these  values. 

Loss  Function 

(ROWE) 

A  function  used  in  decision  theory  for  evaluating  the  losses  incur¬ 
red  when  certain  decisions  are  made  under  uncertainty.  If  the  loss 
function  is  independent  of  the  decision  value  used,  it  is  frequently 
called  a  cost  function. 

Maintainability 

(AF0TECP3) 

Those  characteristics  of  software  which  affect  the  ability  of  the 
software  programmer  to  correct  errors,  enhance  system  capabilities 
through  software  changes,  and  modify  the  software  to  be  compatible 
with  hardware  changes. 

(AF0TECP5) 

The  probability  that  a  system  out  of  service  for  maintenance  can  be 
properly  repaired  and  returned  to  service  in  a  stated  elapsed  time. 

Maintenance  Documentation 

(AF0TECP5) 

The  documentation  that  describes  the  maintenance  of  computer  system 
hardware  and  software. 
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(ROWE) 

a)  Capable  of  being  sensed,  that  which  is  sensed  being  convertible 
to  an  indication;  the  indication  can  be  logical,  axiological,  numer¬ 
ical,  or  probabilistic.  If  probabilistic,  it  is  empirical  and  sub¬ 
jective. 

b)  Comparable  to  some  unit  designated  as  standard. 

Measured  Risk  Level 

(ROWE) 

The  historic,  measured,  or  modeled  risk  associated  with  a  given 
activity. 

Measurement  Uncertainty 
(ROWE) 

The  absence  of  information  about  the  specific  value  of  a  measurable 
variable. 

Methodology 


(ROWE) 

An  open  system  of  procedures. 


Model 


(ROWE) 

An  abstraction  of  reality  that  is  always  an  approximation  to 
reality. 

Module 

(AFR300-15) 

A  program  unit  that  is  discrete  and  identifiable  with  respect  to 
compiling  and  combining  with  other  units. 

Nominal  Scale  (taxonomy) 

(ROWE) 

A  classification  of  items  that  can  be  distinguished  from  one  another 
by  one  or  more  properties. 

Objective  Function 

(ROWE) 

A  specified  mathematical  relationship  between  a  dependent  variable 
(e.g.,  overall  measure  of  benefits)  and  a  set  of  independent  vari¬ 
ables  (e.g.,  individual  benefit  measures  and  their  relative 
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weights).  In  choosing  among  alternatives,  the  decision  maker 
typically  seeks  to  maximize  the  (dependent  variable  of  the)  objec¬ 
tive  function. 

Operational  Effectiveness 

(AF0TECP1) 

The  overall  degree  of  mission  accomplishment  of  a  system  used  by 
representative  personnel  in  the  context  of  the  organization, 
doctrine,  tactics,  threat  (including  countermeasures  and  nuclear 
threats),  and  environment  in  the  planned  operational  employment  of 
the  system.  (OoO  Directive  5000.3) 

Operational  Suitability 

(AF0TECP1) 

The  degree  to  which  a  system  can  be  satisfactorily  placed  in  field 
use,  with  consideration  being  given  availability,  compatibility, 
transportability,  interoperability,  reliability,  wartime  usage 
rates,  maintainability,  safety,  human  factors,  manpower  supportabil- 

ity,  logistic  supportabi 1 ity,  and  training  requirements.  (DoD 

Directive  5000.3) 

Opinion  Survey/Sampling 

(ROWE) 

Any  procedure  for  obtaining  by  oral  or  written  interrogation  or  both 
the  views  of  any  portion  of  the  affected  population  regarding 

benefit  levels  expected,  their  utility,  and/or  relative  importance. 
Typically,  scientific  sampling  procedures  would  be  used  to  maximize 
(for  a  given  level  of  effort)  the  accuracy  and  precision  of  the 

results  obtained. 

Opportunity  Cost 

(ROWE) 

The  value  to  society  of  the  next  best  alternative  use  of  a  resource. 
This  is  the  true  economic  cost  to  society  of  using  a  resource  for  a 
specific  purpose  or  in  a  specific  project. 

Ordinal  Scale  (rank  scale) 

(ROWE) 

An  ordering  (ranking)  of  items  by  the  degree  to  which  they  satisfy 
some  criterion. 

Paradign 

(ROWE) 

A  structured  set  of  concepts,  definitions,  classif ications ,  axioms, 
and  assumptions  used  in  providing  a  conceptual  framework  for  study¬ 
ing  a  given  problem. 
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(ROWE) 

A  technique  for  sensitivity  analysis  of  any  given  model  in  which  the 
values  of  parameters  that  are  input  to  the  model's  calculation  are 
systematically  varied  to  permit  observation  of  how  such  variation 
affects  the  model's  output  (especially  ranking  of  alternatives). 

Personnel 

(AF0TECP5) 

A  general  term  for  the  experience,  education,  and  quantity  of  people 
who  are  assigned  to  the  software  support  facility  either  directly  or 
indirectly  maintaining  the  ECS.  It  includes  Management,  Technical, 
Support,  and  Contractor  resources. 

Personnel  Profile 

(AF0TECP5) 

The  characteristics  that  describe  the  experience,  education,  and 
quantity  of  software  support  facility  personnel. 

Physical  Configuration  Audit  (PCA) 

(AFR300- 15) 

The  formal  examination  of  the  coded  version  of  a  computer  program 
configuration  item  against  its  technical  documentation. 

Precision 

(ROWE) 

The  exactness  with  which  a  quantity  is  stated,  that  is,  the  number 
of  units  into  which  a  measurement  scale  of  that  quantity  may  be 
meaningfully  divided.  The  number  of  significant  digits  is  a  measure 
of  precision. 

Predictive  Model ing 

(ROWE) 

Use  of  any  mathematic  model  that  estimates  or  predicts  the  value  of 
a  dependent  variable  in  terms  of  component  factors  specified  as 
independent  variables. 

Preference 

(ROWE) 

Assignment  of  rank  to  items  by  an  agent  when  the  criterion  used  is 
utility  to  the  ranking  agent. 
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Probabi 1 ity 
(ROWE) 

A  numerical  property  attached  to  an  activity  or  event  whereby  the 
likelihood  of  its  future  occurrence  is  expressed  or  clarified. 

Probability  Distribution 

(ROWE) 

The  representation  of  a  repeatable  stochastic  process  by  a  function 
satisfying  the  axioms  of  probability  theory. 

Probability  of  Occurrence 

(ROWE) 

The  probability  that  a  particular  event  will  occur,  or  will  occur  in 
a  given  interval. 

Probability  Threshold 

(ROWE) 

A  probability  of  occurrence  level  for  a  risk  below  which  a  risk 
agent  is  no  longer  concerned  with  the  risk  and  ignores  it  in  prac¬ 
tice  (Threshold  of  concern). 

Product  Baseline 

(AFR300-15) 

The  initial  approved  product  configuration  identification. 

Product  Verification  Review  (PVR) 

(AFR300-15) 

A  formal  review  conducted  by  the  developer  for  each  CPCI  at  the  end 
of  the  development  phase  to  establish  the  Product  Baseline  for  that 
CPCI  and  to  ensure  preparation  for  the  Test  Phase  has  been  com¬ 
pleted. 

Program  Manager 
(AFR800-14) 

The  generic  term  used  to  denote  a  single  Air  Force  manager  (System 
Program  Director,  Program/Project  Manager,  or  System/Item  Manager) 
during  any  specific  phase  of  the  acquisition  life  cycle. 
(AFR  800-2). 

Program  Management  Directive  (PMD) 

(AFR800-14) 

The  official  HQ  USAF  management  directive  used  to  provide  direction 
to  the  implementing  and  participating  commands  and  satisfy  documen¬ 
tation  requirements .  It  will  be  used  during  the  entire  acquisition 
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cycle  to  state  requirements  and  request  studies  as  well  as  initiate, 
approve,  change,  transition,  modify  or  terminate  programs.  The  con¬ 
tent  of  the  PMD,  including  the  required  HQ  USAF  review  and  approval 
actions,  is  tailored  to  the  needs  of  each  individual  program. 
(AFR  800-2) 

Program  Management  Plan  (PMP) 

(AFR800-14) 

The  document  developed  and  issued  by  the  Program  Manager  which  shows 
the  integrated  time-phased  tasks  and  resources  required  to  complete 
the  task  specified  in  the  PMD.  The  PMP  is  tailored  to  the  needs  of 
each  individual  program.  (AFR  800-2) 

Program  Office  (P0) 

(AFR800-14) 

The  field  office  organized  by  the  Program  Manager  to  assist  him  in 
accomplishing  the  program  tasks.  (AFR  800-2) 

Program  Support  Tools 

(AF0TECP3) 

General  debug  aids,  test/retest  software,  trace  software/hardware 
features,  use  of  compiler/1  ink  editor,  library  management/configura¬ 
tion  management/text  editor/display  software  tools. 

Program  Test  Plan 

(AF0TECP3) 

Set  of  descriptions  and  procedures  for  how  the  program  is  to  be  (or 
can  be,  or  has  been)  tested. 

Propensity  for  Risk  Acceptance 

(ROWE) 

An  individual,  subjective  trait  designating  the  degree  of  risk  one 
is  willing  to  subject  himself  to  for  a  particular  purpose. 

Quality  Assurance  (QA) 

(AFR300-15) 

All  actions  that  are  taken  to  assure  that  a  development  organization 
delivers  products  that  meet  performance  requirements  and  adhere  to 
standards  and  procedures. 

Quantification 

(ROWE) 

The  assignment  of  a  number  to  an  entity  or  a  method  for  determining 
a  number  to  be  assigned  to  an  entity 
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Rel iabi 1  ity 
(ROWE) 

The  probability  that  the  system  will  perform  its  required  functions 
under  given  conditions  for  a  specified  operating  time. 

Residual  Risk 

(AFR205X) 

That  portion  of  risk  which  remains  after  security  measures  have  been 
applied . 

Risk 

(AFR205X) 

The  loss  potential  which  exists  as  the  result  of  threat/vulnerabil¬ 
ity  pairs.  Reducing  either  the  threat  or  the  vulnerability  reduces 
the  risk. 

(ROWE) 

The  potential  for  realization  of  unwanted,  negative  consequences  of 
an  event. 

Risk  Acceptance 

(ROWE) 

Willingness  of  an  individual,  group,  or  society  to  accept  a  specific 
level  of  risk  to  obtain  some  gain  or  benefit. 

Risk  Acceptance  Function 

(ROWE) 

A  subjective  operator  relating  the  levels  of  probability  of  occur¬ 
rence  and  value  of  a  consequence  to  a  level  of  risk  acceptance. 

Risk  Acceptance  Level 

(ROWE) 

The  acceptable  probability  of  occurrence  of  a  specific  consequence 
value  to  a  given  risk  agent. 

Risk  Acceptance  Utility  Function 

(ROWE) 

The  profile  of  the  acceptability  of  the  probability  of  occurrence 
for  all  consequences  involved  in  a  risk  situation  for  a  specific 
risk  agent. 

Risk  Agent 

(ROWE) 

See  Valuing  Agent. 
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Risk  Analysis 
(AFR205X) 

A  part  of  risk  management  that  is  used  to  minimize  risk  by  effec¬ 
tively  applying  security  measures  commensurate  with  the  relative 
threats,  vulnerabilities,  and  values  of  the  resources  to  be 
protected.  (The  value  of  the  resources  includes  impact  on  the 
organizations  the  automatic  data  processing  system  supports,  and 
impact  of  the  loss  or  unauthorized  modification  of  data).  Risk 
analysis  may  be  thought  of  as  consisting  of  four  modules:  sensitiv¬ 
ity  assessment,  risk  assessment,  economic  assessment,  and  security 
test  and  evaluation. 

Risk  Assessment 

(AFR2Q5X) 

A  detailed  study  of  the  vulnerabilities,  threats,  likelihood,  loss 
or  impact,  and  theoretical  effecti veness  of  security  measures.  The 
results  of  a  risk  assessment  may  be  used  to  develop  security 
requirements  and  specifications. 

(ROWE) 

The  total  process  of  quantifying  a  risk  and  finding  an  acceptable 
level  of  that  risk  for  an  individual,  group,  or  society.  It 
involves  both  risk  determination  and  risk  evaluation. 

Risk  Averse 

(ROWE) 

Displaying  a  propensity  against  taking  risks. 

Risk  Aversion 
(ROWE) 

The  act  of  reducing  risk. 

Risk  Baseline 
(CURRENT) 

The  risk  probability  density  function  and  the  associated  magnitude 
of  consequence  for  the  potential  negative  outcomes. 

Risk  Consequence 

(ROWE) 

The  impact  to  a  risk  agent  of  exposure  to  a  risky  event. 

Risk  Conversion  Factor 
(ROWE) 

A  numerical  weight  allowing  one  type  of  risk  to  be  compared  to 
another  type. 
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Risk  Determination 
(ROWE) 

The  process  of  identifying  and  estimating  the  magnitude  of  risk. 

Risk  Estimation 
(ROWE) 

The  process  of  quantification  of  the  probabilities  and  consequence 
values  for  an  identified  risk. 

Risk  Evaluation 

(ROWE) 

The  complex  process  of  developing  acceptable  levels  of  risk  to 
individuals  or  society. 

Risk  Evaluator 

(ROWE) 

A  person,  group,  or  institution  that  seeks  to  interpret  a  valuing 
agent's  risk  for  a  particular  purpose. 

Risk  Identification 

(ROWE) 

The  observation  and  recognition  of  new  risk  parameters,  or  new 
relationships  among  existing  risk  parameters,  or  perception  of  a 
change  in  the  magnitude  of  existing  risk  parameters. 

Risk  Management 

(AFR205X) 

The  total  process  of  identifying,  controlling,  and  minimizing 
uncertain  events.  The  process  of  obtaining  and  maintaining  DAA 
approval  is  a  major  element  of  the  risk  management  program.  The 
process  facilitates  the  management  of  automatic  data  processing 
(ADP)  security  risks  by  each  level  of  ADP  management  throughout  the 
ADP  life  cycle.  The  approval  process  consists  of  three  elements: 
risk  analysis,  certification,  and  approval. 

Risk  Profile  Baseline 

(CURRENT) 

The  measure  of  information  and/or  requirements  which  serve  as  the 
zero  reference  against  which  negative  (and  positive)  outcomes  can  be 
determined. 
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Risk  Proportionality  Derating  Factor 
(ROWE) 

Quantifying  the  degree  to  which  risks  become  less  acceptable  as 
indirect  benefits  to  the  risk  agent  declines. 

Risk  Proportional ity  Factor 

(ROWE) 

That  portion  of  the  total  societal  risk  that  society  will  accept  for 
a  new  technology. 

Risk  Reduction 

(ROWE) 

The  action  of  lowering  the  probability  of  occurrence  and/or  the 
value  of  a  risk  consequence,  thereby  reducing  the  magnitude  of  the 
risk. 

Risk  Reference 
(ROWE) 

Some  reference,  absolute  or  relative,  against  which  the  acceptabil¬ 
ity  of  a  similar  risk  may  be  measured  or  related;  implies  some 
overall  value  of  risk  to  society.  » 

Risk  Referent 

(ROWE) 

A  specific  level  of  risk  deemed  acceptable  by  society  or  a  risk 
evaluator  for  a  specific  risk;  it  is  derived  from  a  risk  reference. 

Risky  Shift 

(ROWE) 

The  tendency  of  certain  groups  to  become  more  extreme  or  take 
riskier  positions  in  their  judgments  than  they  would,  acting  as 
individuals. 

Sensitivity  Analysis 

(ROWE) 

A  method  used  to  examine  the  operation  of  a  system  by  measuring  the 
deviation  of  its  nominal  behavior  due  to  perturbations  in  the  per¬ 
formance  of  its  components  from  their  nominal  values. 

Simul ation 

(AFR800-14) 

The  representation  of  physical  systems  or  phenomena  by  computers, 
models  or  other  equipment. 
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Software 

(AF0TECP1) 

A  set  of  computer  programs,  procedures,  and  associated  documentation 
concerned  with  the  operation  of  a  data  processing  system. 

(CURRENT) 

The  programs  which  execute  in  a  computer.  The  data  input,  output, 
controls  upon  which  program  execution  depends  and  the  documentation 
which  describes,  in  a  textual  medium,  development  and  maintenance  of 
the  programs. 

Software  Error 

(CURRENT) 

The  human  decision  (inadvertent  or  by  design)  which  results  in  the 
inclusion  of  a  fault  in  a  software  product. 

Software  Fault 

(CURRENT) 

The  presence  or  absence  of  that  part  of  a  software  product  which  can 
result  in  software  failure. 

Software  Maintainability 

(AF0TECP1) 

The  ease  with  which  software  can  be  changed  in  order  to: 

(1)  Correct  errors. 

(2)  Add  or  modify  system  capabilities  through  software 
changes . 

(3)  Delete  features  from  programs. 

(4)  Modify  software  to  be  compatible  with  hardware  changes. 
(CURRENT) 

A  quality  of  software  which  reflects  the  effort  required  to  perform 
software  maintenance  actions. 

Software  Maintenance 

(CURRENT) 

Those  actions  required  for: 

(1)  Correction.  Removal,  correction  of  software  faults 

(2)  Enhancement.  Addition/deletion  of  features  from  the 
software 

(3)  Conversion.  Modification  of  the  software  because  of 
environment  ('data  hardware)  changes. 
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Software  Maintenance  Environment 
(CURRENT) 

An  integration  of  personnel  support  systems  and  physical  facilities 
for  the  purpose  of  maintaining  software  products. 

Software  Maintenance  Measures 

(CURRENT) 

Measures  of  software  maintainability  and  environment  capabilities  to 
support  software  maintenance  activity. 

Software  Management 

(CURRENT) 

The  policy,  methodology,  procedures,  and  guidelines  applied  in  a 
software  environment  to  the  software  development/maintenance  activi¬ 
ties.  Also,  those  personnel  with  software  management  responsi¬ 
bilities. 

Software  Reliability 
(CURRENT) 

A  quality  of  software  which  reflects,  the  probability  of  failure  free 
operation  of . a  software  component  or  system  in  a  specified  environ¬ 
ment  for  a  specified  time. 

Software  Portability 

(CURRENT) 

A  quality  of  software  which  reflects  the  effort  required  to  transfer 
the  software  from  one  environment  (hardware  and  system  software)  to 
another. 

Software  Support  Facility  (SSF) 

(AF0TECP5) 

The  Facility  which  houses  and  provides  services  for  the  Support 
Systems  and  Personnel  required  to  maintain  the  software  for  a 
specific  ECS. 

Software  Supportability 

(CURRENT) 

A  measure  of  the  adequacy  of  personnel,  resources,  and  procedures  to 
facilitate:  - 

(1)  Modifying  and  installing  software 

(2)  Establishing  an  operational  software  baseline 

(3)  Meeting  user  requirements. 
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Specification 

(AFR300-15) 

A  document  that  describes  the  requirements  for  the  development  or 
acquisition  of  AOPE  and/or  software. 

Standards 

(AF0TECP3) 

Procedures,  rules,  and  conventions  used  for  prescribing  disciplined 
program  design  and  implementation. 

States  of  Nature 
(ROWE) 

A  concept  from  decision  theory.  In  decision  making  under  uncer¬ 
tainty,  the  outcomes  (numerical  results)  associated  with  each  avail¬ 
able  alternative  are  considered  to  be  predictable  as  a  set  of  n  dis¬ 
crete  values  depending  on  conditions  beyond  the  decision  maker's 
control  and  for  which  he  has  no  useful  estimates  of  the  respective 
probabilities.  The  n  sets  of  conditions  under  which  each  one  of  the 
outcomes  is  expected  are  termed  "states  of  nature." 

Structured  Value  (structured  value  analysis) 

(ROWE) 

The  resultant  value  of  a  particular  value  set  evaluated  for  a  par¬ 
ticular  data  set.  This  value  lies  between  zero  and  unity  and  allows 
many  data  sets  to  be  ranked  numerically  to  relation  to  one  another. 

Structured  Value  Analysis 
(ROWE) 

A  multistage  procedure  for  assessing  the  value  of  an  action,  project 
alternative,  and  so  on,  incorporating  individual  techniques  at  each 
stage  for  computing  from  quantitative  measures  of  individual  com¬ 
ponents  a  single  figure  expressing  the  overall  value.  A  multistage 
procedure  for  assessing  the  value  of  an  action,  project,  alterna¬ 
tive,  and  so  on,  by  structuring  the  complete  entity  into  component 
elements,  to  each  of  which  a  numeric  measure  of  value  (positive  or 
negative)  can  be  assigned.  These  are  then  coverted  to  a  common 
utility  scale.  Each  component  is  assigned  a  weight  expressing  its 
relative  significance  in  determining  overall  value  of  the  entity.  A 
single  figure  of  worth  or  value  is  then  computed  from  measures  and 
weights  of  all  individual  components.  The  procedure  permits  con¬ 
siderable  flexibility  in  choice  of  techniques  used  to  perform  each 
necessary  optimal  step. 
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Subjective  Probabilities 
(ROWE) 

The  assignment  of  subjective  weights  to  possible  outcomes  of  an 
uncertain  event  where  weights  assigned  satisfy  axioms  of  probability 
theory. 

Support  Personnel 
(AF0TECP5) 

A  general  term  for  military  or  DoD  civilian  personnel  whose  skills 
are  necessary  for  the  software  support  facility  to  function  but  who 
do  not  directly  support  ECS  software  maintenance. 

Support  System 

(AF0TECP5) 

Any  automated  system  used  to  change,  test,  or  manage  the  configur¬ 
ation  of  ECS  software  and  associated  documentation.  Includes  but  is 
not  limited  to  Host  Processor,  Software  Bench,  Laboratory-Integrated 
Test  Facility,  Operational-Integrated  Test  Facility,  and  Configura¬ 
tion  Management  System. 

Support  System  Facility 

(AF0TECP5) 

The  facility  resources  that  must  be  available  for  the  software 
support  resources  to  accomplish  a  specific  task(s)  (see  General 
Facility). 

Surrogate  or  Proxy  Measures 
(ROWE) 

The  use  of  a  related  quantity  as  a  proxy  for  an  unknown  or  diffi- 
cult-to-measure  value.  The  relationship  may  be  established  by  arm¬ 
chair  analysis,  correlation  techniques,  scientific  studies,  or  other 
means . 

System 

(ROWE) 

a)  A  complex  entity  formed  of  many,  often  diverse,  parts  subject  to 
a  conmon  plan  or  serving  a  common  purpose. 

b)  A  composite  of  equipment,  skills,  and  techniques  capable  of  per¬ 
forming  and/or  supporting  an  operation. 

System  Design  Review  (SDR) 

(AFR300-15) 

A  formal  review  of  the  system  design  approach  for  an  ADS. 
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System  Requirements  Review  (SRR) 

(AFR300-15) 

A  formal  review  of  the  requirements  for  an  ADS. 

System  Software 
(AF0TECP5) 

All  of  the  software  that  is  part  of  the  software  support  facility 
computer  system.  It  is  never  or  seldom  accessed  directly  by  soft¬ 
ware  support  facility  personnel;  it  controls  the  processing  of 
application  software.  It  includes  the  Operating  System,  Source  Code 
Editor,  Language  Translator,  Link  Editor/Loader,  Librarian/File 
Manager,  Data  Base  Manager,  and  Automated  Software  Development  Tool. 

Taxonomy 

(ROWE) 

The  identification  and  definition  of  properties  of  elements  of  the 
universe;  a  disaggregation,  as  contrasted  with  systematics  (which  is 
an  aggregation)  and  as  contrasted  with  morphology  (which  encompasses 
both  taxonomy  and  systematics). 

Test  Analysis  Report  (RT) 

(AFR300-15) 

A  document  containing  the  results  and  analyses  of  tests  executed 
during  the  Test  Phase. 

Threshold 

(ROWE) 

A  discontinuous  change  of  state  of  a  parameter  as  its  measure 
increases.  One  condition  exists  below  the  discontinuity,  and  a 
different  one  above  it. 

Transfer 

(AFR800-14) 

That  point  in  time  when  the  designated  Supporting  Command  accepts 
program  management  responsibilities  from  the  Implementing  Command. 
This  includes  logistic  support  and  related  engineering  and  procure¬ 
ment  responsibilities.  (AFR  800-4) 

Turnover 

(AFR800-14) 

That  point  in  time  when  the  operating  command  formally  accepts 
responsibility  from  the  Implementing  Comnand  for  the  operation  and 
maintenance  of  the  system,  equipment,  or  computer  program  acquired. 
(AFR  800-19) 


wiiiiiiuuiuiuijimuiuvM 


THE  BDM  CORPORATION 


BDM/A-84-496-TR 


Uncertainty 


(ROWE) 

The  absence  of  information;  that  which  is  unknown. 


User 


(AFR205X ) 

Any  persons  (or  organizations)  having  access  to  an  automatic  data 
processing  system  via  communication  through  a  remote  device  or  who 
is  allowed  to  submit  input  to  the  system  through  other  media  (for 
example,  tape  or  card  decks).  (Does  not  include  those  persons  or 
organizations  defined  as  customers.) 


Valuation 


(ROWE) 

The  act  of  mapping  an  ordinal  scale  onto  an  interval  scale  (i.e., 
assigning  a  numerical  measure  to  each  ranked  item  based  on  its 
relative  distance  from  the  end  points  of  the  interval  scale... 
assigning  an  interval  scale  value  to  a  risk  consequence. 


Value 


(ROWE) 

A  quality  quantified  on  a  scale  expressing  the  satisfaction  of  man's 
intrinsic  wants  and  desires. 


Value  Function  (structured  value  analysis) 


(ROWE) 

A  function  relating  points  on  the  parameter  measurement  scale  to  the 
value  scale  for  a  particular  parameter.  These  functions  may  result 
from  explict  information  or  may  be  arrived  at  through  value  judg¬ 
ment. 


Value  Set  (structured  value  analysis) 


(ROWE) 

A  specific  set  of  model  parameters  made  up  of  terms  and  factors, 
expressed  in  particular  measurement  scales,  value  functions,  and 
weights . 


Valuing 


(ROWE) 

The  act  of  assigning  a  value  to  a  risk  consequence. 
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Valuing  Agent 
(ROWE) 

A  person  or  group  of  persons  who  evaluates  directly  the  consequence 
of  a  risk  to  which  he  is  subjected.  A  risk  agent. 

Verification/Validation  (of  computer  programs) 

(AFR800-14) 

The  process  of  determining  that  the  computer  program  was  developed 
in  accordance  with  the  stated  specification  and  satisfactorily  per¬ 
forms,  in  the  mission  environment,  the  function(s)  for  which  it  was 
designed. 

Weight  (structured  value  analysis) 

(ROWE) 

The  relative  importance  of  terms  in  a  model  expressed  as  a  decimal 
fraction;  weights  for  a  set  of  terms  add  to  unity. 

Weighting  Factor 

(ROWE) 

A  coefficient  used  to  adjust  variable  accuracy  to  a  subjective 
evaluation;  these  factors  are  usually  determined  through  surveys, 
Delphi  sessions,  or  other  formats  of  expressing  social  priorities. 
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APPENDIX  C 
POLICY  DIRECTIVES 


C.l  GENERAL. 

This  appendix  summarizes  and  annotates  the  directives  of  higher 
authorities  and  the  military  services.  Material  in  this  appendix  is 
derived  from  reference  5.35. 

These  directives  were  identified  by  reviewing  the  system  acquisition 
and  management  directives  of  0MB,  DoD,  and  the  Air  Force,  and  noting  all 
references  which  deal  specifically  with  risk  analysis.  Some  references 
imply  the  need  for  risk  analysis,-  but  do  not  explicitly  state  such 
requirement.  Although  additional  documents  were  reviewed,  only  those 
listed  were  found  to  contain  material  relevant  to  risk  analysis. 

C  .2  HIGHER  LEVEL  REQUIREMENTS. 

This  section  lists  excerpts  and  comments  briefly  on  0MB  and  DoD 
policies  and  directives  relating  to  risk  assessment. 

C.2.1  Office  of  Management  and  Budget. 

0 MB  Circular  A-109.  Major  System  Acquisition  (5  April  1976). 
Paragraph  7.  "Each  agency  acquiring  major  systems  should...  tailor 
an  acquisition  strategy  for  each  program.  ...The  strategy  should 
typically  include. . .methods  for  analyzing  and  evaluating  contractor  and 
Government  risks." 

C.2.2  Department  of  Defense. 

DoD  Directive  (DoDD)  5000.1.  Major  System  Acquisition  (29  March 
1982). 

Paragraph  C.2.C.(3).  To  achieve  program  stability,  DoD  components 
will  "estimate  and  budget  realistically,  and  fund  adequately,  procurement 
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(research,  develooment,  and  Droduction),  logistics  and  manoower  for  major 
systems." 

Paragraph  £.4.C.(l)(a) .  This  Daragraph  states  that  it  may  be 
reasonable  to  delay  Milestone  II  decisions  until  some  develooment  efforts 
are  accomplished  in  order  to  "reduce  risk  and  uncertainty  before  the 
commitment  to  a  major  increase  in  the  application  of  resources  toward 
full-scale  development  is  made." 

Paragraph  E.3.  "Coirmensurate  with  risk,  such  approaches  (to  reduce 
acguisition  time)  as  develoDing  separate  alternatives  in  high-risk  areas, 
should  be  encouraged." 

DoDI  5000.2  Major  Systems  Acguisition  Procedures  (March  3,  1933). 

No  references  to  risk  analysis  appear  in  the  body  of  the  text, 
however,  paragraphs  D.3.e.(l)(a)  and  D.3.e.(2)(a)  refer  to  the  need  for 
System  Concept  Paoers  (SCP's)  and  Oecision  Coordinating  Papers  (DCP's)  to 
establish  and  identify  goals,  thresholds,  and  threshold  ranges  (emphasis 
supplied),  thus  recognizing  the  concept  of  risk. 

Enclosure  (4)  Format  for  SCP  and  DCP. 

"VIII.  Technological  Risks  of  Selected  Alternative.  For  Mile¬ 
stone  I  (SCP),  identify  key  areas  of  technological  risk  which  must  be 
reduced  by  R&D  and  validated  by  T&E  before  Milestone  II.  For  Mile¬ 
stone  II  (DCP),  discuss  T&E  results  that  show  all  significant  risk  areas 
have  been  resolved.  Also,  for  Milestone  II,  verify  that  technology  is  in 
hand  and  also  engineering  (rather  than  experimental)  effort  remains." 

OoDI  5000.38.  Production  Readiness  Reviews  (24  January  1979). 

Paragraph  A. 2.  "The  objective  of  a  Program  Readiness  Review  (PRR) 
is  to  verify  that  the  production,  design,  planning,  and  associated 
preparations  for  a  system  have  progressed  to  the  point  where  a  Droduction 
commitment  can  be  made  without  incurring  unacceptable  risks  of  breaching 
thresholds  of  schedule,  performance,  cost,  or  other  established 
criteria." 

Paragraph  E.4.  "The  OPESO  (DoD  Product  Engineering  Services  Office) 
independent  production  readiness  assessment  will  consist  of  objective 
conclusions  based  on  the  findings  of  the  PRR  and  other  investigations. 
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This  assessment  will  identify  potential  problem  areas  which  constitute 
production,  cost,  or  schedule  risks.  Each  risk  will  be  expressed  in 
terms  of  its  relative  magnitude  and  potential  consequences."  (Emphasis 
supplied. ) 

DoOI  7041.3.  Economic  Analyses  and  Program  Evaluation  for  Resource 
Management  (October  19,  1972). 

Enclosure  (2) 

Paragraph  3.7.  "Risk/Uncertainty  Analysis.  Risk  assessments  will 
be  made  to  determine  the  expectation  or  probability  that  proqram/project 
objectives  will  be  realized  by  following  a  specific  course  of  action  with 
constraints  of  time,  cost,  and  technical  performance.  (Emphasis  sup¬ 
plied.)  Actual  costs  and  outputs  of  many  OoD  projects  differ  from  those 
expected  at  the  time  of  decision.  For  those  cases,  and  in  particular  for 
major  weapon  systems  covered  by  a  Selected  Acquisition  Review  Report  or 
subject  to  review  by  the  Defense  System  Acquisition  Review  Committee 
(DSARC),  the  impact  which  could  result  from  this  variability  should  be 
evaluated." 

Paragraph  B.7.a.  "Independent  parametric  cost  estimates  can  provide 
an  early  test  of  the  reasonableness  of  cost  estimates.  Independent 
parametric  cost  estimates  will  be  made  at  key  decision  points  for  major 
weapon  systems,  e.g.,  during  concept  formulation  and  prior  to  making 
major  coimitments  of  funds  for  development  and  production.  These  esti¬ 
mates  generally  consider  cost  at  high  levels  of  aggregation  and  are 
predicated  on  actual  historical  costs  encountered  in  like  or  similar 
programs.  As  such,  they  incorporate  costs  for  expected  uncertainties  on 
the  average.  (1)  Costs  should  be  derived  by  parametric  techniques  and 
expressed  as  feasible  ranges  in  terms  of  the  parameters  which  drive  them. 
It  is  most  important  that  estimates  be  presented  as  cost  ranges  related 
to  the  probable  values  of  system  parameters,  characteristics,  or  attrib¬ 
utes  which  are  determined  by  costs.  (Emphasis  supplied).  (2)  These 
estimates  will  be  available  for  each  DSARC  review.  Parametric  estimates 
will  be  derived  independent  of  functional,  program  manager  or  contractor 
influence.  (3)  When  the  independent  parametric  cost  estimate 
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differs  from  the  program  manager's  current  estimate,  the  latter  estimate 
will  be  used  for  economic  analysis/orogram  evaluations.  Once  a  orogram 
estimate  is  established  as  a  baseline,  a  program/project  manager  will 
manage  his  program  within  that  limitation.  (4)  The  orogram  manager's 
current  estimate  will  be  an  assessment  of  the  ultimate  cost  expected  for 
a  program/project  including  undef initi?ed  contingencies.  (Emphasis 
suDplied.)  As  such,  the  program  manager's  current  estimate  should  be 
relatively  stable  over  long  periods  of  time  and  not  change  with  small 
incremental  changes  to  the  approved  program,  funding  changes,  or  finan¬ 
cial  fluctuations.  To  the  extent  possible,  schedules,  and  funding  should 
be  structured  to  accommodate  program  uncertainties  and  unforeseen  prob¬ 
lems.  11  (Emphasis  supplied.) 

Paragraph  8.7.b.  "Special  degrees  of  risk/uncertainty  associated 
with  a  particular  program/project,  may  be  pointed  out  quantitatively  in 
an  analysis  and  used  for  program  review  purposes.  Probability  estimates 
can  be  developed  by  testing  the  sensitivity  of  key  variables  on  estimated 
costs  and  performance.  The  probability  that  each  of  the  possible  cost  or 
output  estimates  may  be  realized  should  be  discussed  narratively  when 
there  is  no  basis  for  a  quantitative  estimate."  (Emphasis  supplied.) 

Paragraph  B.7.c.  Estimates  will  be  expressed  in  terms  of  perform¬ 
ance  thresholds,  goals,  or  ranges.  Program/oroject  estimates  will 
include  the  limits  within  which  ultimate  program  cost  and  technical 
performance  is  expected  to  fall." 

C.3  SERVICE  REQUIREMENTS  (U.S.  AIR  FORCE). 

The  following  Air  Force  directives  address  consideration  of  program 
risk  as  revealed  by  excerpt  or  editorial  summation. 

C.3.1  Air  Force  Regulation  AFR  173-11. 

Independent  Cost  Analysis  Program  (12  Dec  1980). 

Paragraph  5.  Definition  and  Scope  of  the  Independent  Cost  Analysis 
(ICA). 
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Paragraph  6.1.  "Will  contain  a  detailed  risk  assessment  to  include 
risk  related  to  the  cost  estimating  techniques  employed  and  with  tech¬ 
nical  and  schedule  uncertainties  that  may  have  an  impact  on  cost  esti¬ 
mates.  It  will  also  include  sensitivity  analyses  of  critical  assumptions 
ana  cost  driving  parameters." 

Paragraph  7,e.  "For  cost  elements  with  a  high  degree  of  uncer¬ 
tainty,  the  ICA  will  provide  sensitivity  analysis  using  frequency  distri¬ 
butions  or  ranges  of  cost.  The  probability  distributions  used  to  prepare 
range  estimates,  as  well  as  the  proper  assumptions,  must  be  Drovided. 
"Prediction  intervals  around  cost  estimating  relationships  ^CERs'1  or 
Monte  Carlo  simulations  will  be  used  as  proper  in  quantifying  risk. " 
(Emphasis  supplied.) 

Paragraph  9.d.  "The  I  SR  will  address  the  potential  risk  in  the 
program  office  estimate  by  identifying  'risk'  areas  and  their  probable 
and  possible  cost  impact."  (ISR  means  independent  schedule  review.) 

C.3.2  AFR  70-15.  Source  Selection  Policy  and  Procedures. 

Paragraph  1-4. d.  "The  source  selection  process  shall  focus  adequate 
attention  on  the  program  risk  and  uncertainties  during  solicitation, 
proposed  evaluation,  and  selection  phases. 

a)  Offerors  should  not  be  penalized  for  the  identification  of 
risk  associated  with  their  prooosals.  Proposals  should  be 
credited  when  realistic  approaches  for  risk  resolution  are 
provided. 

b)  The  procuring  activity  shall  prepare  an  independent  risk 
assessment  before  receipt  of  proposals,  to  facilitate  risk 
analysis  evaluation. 

Paragraph  2-2. c(3).  "It  (the  evaluation  criteria)  must  address 
those  high  risks  and  technical  uncertainties,  which  were  identified  by 
the  offerors  and  the  Government  as  'known-unknowns’  during  the  conceptual 
phase.  An  indication  should  also  be  provided  of  the  relative  importance 
of  each  criterion  for  later  use  in  the  solicitation." 
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°araqraph  2-4. d.  "...Risk  analysis  is  a  Dart  of  the  evaluation 
process,  and  risk  assessment  for  each  proposal  must  be  included  in  all 
reports  to  the  Source  Selection  Advisory  Council  (SSAC)  and  Source 
Selection  Authority  (SSA).  Technical  risk,  as  pertains  to  each  proposal, 
should  be  rated  based  on  the  offeror's  risk  assessment  and  the  credibil¬ 
ity  of  his  proposed  approach  for  eliminating  or  avoiding  such  risks." 

Paragraph  3-2. a. (2).  "The  solicitation  should. .. include  a 
discussion  of  known  or  potential  risks,  where  there  is  reason  to  believe 
that  the  potential  offerors  are  not  aware  of  the  risks." 

Paragraph  3-7. e.  "The  offerors  must  be  required  to  submit  a  risk 
analysis  as  part  of  their  proposal  which  also  identifies  risk  areas  and 
which  furnishes  an  insight  to  the  evaluator  as  to  how  the  offeror  intends 
to  resolve  these  risks  and  the  alternatives  to  overcoming  the  high  risk 
approaches.  In  order  to  aid  the  evaluator  in  performing  the  risk  anal¬ 
ysis,  the  procuring  activity  should  prepare  an  independent  risk  assess¬ 
ment  prior  to  receipt  of  proposals." 

Paragraph  3-8. b. (5).  This  paragraph  states  that  the  SSA  must 
determine  cost/price  risk  inherent  in  each  proposal. 

Attachment  4  -  VIII.  Risk  Analysis. 

This  paragraph  lists  risk  analysis  documentation  format. 

C.3.3  AFR  S00-3.  Engineering  for  Defense  Systems,  f 17  June  19771. 

Paragraph  4.b.  (In  the  validation  phase) ".. .certain  technical 
aspects  may  need  to  be  intensified,  such  as  technical  and  cost  risk 
reduction,  obtaining  a  best  mix  of  technical  requirements,  and  other 
considerations  or  thresholds  as  may  be  described  in  the  P^D." 

Paragraph  5.f.  The  AFSC  "programs  their  research  and  development 
(R&O)  projects  to  develop  and  improve  systems  engineering  methods  and 
techniques  (system  cost  effectiveness,  risk  assessment,  technical  per¬ 
formance  measurement,  etc.)." 


C-6 


THE  BDM  CORPORATION 


3DM/A-34-496-TR 


C.3.4  APR  800-3.  ILS  Program,  (7  February  1980). 

Paragraph  5.r.  "Risk  analysis  and  assessment  and  tradeoff  analyses 
will  be  conducted,  using  the  latest  data  available." 


C.3.5  APR  300-9.  Manufacturing  Management  for  Air  Force  Acquisitions 
(1  October  1979). 


Paragraph  2.c.  "In  the  manufacturing  assessment  of  system  and 
design  alternatives  the  program  manager  will:  (1)  consider  the  relation¬ 
ship  between  several  factors  (such  as  producibi lity,  manufacturing  risks, 
productivity,  ...)  and  evaluate  their  impact  on  the  minimum  essential 
performance  reguirements. " 

Attachment  1.  Extracts  from  OoOO  5000.34,  31  October  1977. 

Paragraph  D.5.  "...Production  risks,  which  should  be  identified  as 
early  as  possible  in  the  acguisition  cycle,  shall  be  reduced  to  acceot- 
able  levels  prior  to  production  decision." 


C.3.6  Aeronautical  Systems  Division  Regulation  (ASDR)  173-1  Aeronautical 
Systems  Division  Cost  Analysis  Program  f 21  October  1981). 


Attachment  5:  Cost  Estimate  Risk  Assessment  Guidelines. 

Paragraph  1.  "...The  purpose  of  the  risk  analysis  described  below 

is  to  alert  decision  makers  to: 

a)  Inputs  or  assumptions  where  a  percent  change  in  an  input 
or  assumption  value  would  make  at  least  one  half  that 
percent  change  in  the  total  estimate. 

b)  Areas  of  uncertainty  at  the  time  the  estimate  was 
prepared. . . 

Paragraph  2.  "Generally,  risk  assessments  must  be  prepared  so  that 
while  not  all  possible  areas  of  risk  are  addressed,  the  overall  amount  of 
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risk  of  cost  growth  can  be  addressed  by  review  of  the  several  highest 
risk  areas  identified  and  discussed." 

Paragraph  3.  "The  risks  of  cost  increase  over  the  estimates  to  be 
addressed  will  be  primarily  those  associated  with  estimating  methods  are 
(sic)  available  data/information  limitations.  The  risks  of  strikes, 
major  test  or  technical  aDproach  failures,  directed  program  changes, 
etc.,  are  not  to  be  addressed." 
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